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SURVEILLANCE OF INFORMATION TECHNOLOGY 
PERFORMANCE-BASED SERVICE ACQUISITIONS (PBSAS) 


ABSTRACT 


There has been an increased emphasis on utilization Perfonnance Based Services 
Acquisitions (PBSA) as a means for organizations to cut costs and to become more 
efficient. According to GAO report 05-274 nearly half of an organization’s yearly budget 
is being spent on services. Consequently, DoD must ensure that contractors perfonning 
these services are being effectively monitored. This will ensure that required services are 
being performed according to contract terms. Based on this GAO report, DoD is not 
adequately performing contractor surveillance. (GAO Report No. 05-274, 2005) 

This can be accomplished by the establishing a PBSA team from the onset of the 
acquisition and ensuring that the technical team members also perform the contractor 
surveillance. To join the team, each person must have received training on PBSA as a 
condition of membership. Furthermore, team members should have access to lesson 
learned from previous and/or current PBSA experiences. (Seven Steps to PBSA, 2005) 

In addition, the PBSA should ensure that FAR 1.102 guiding principles are used 
when developing as well as monitoring contractor performance. (FAR 1.102) 

The purpose of this joint applied project is to examine IT performance based 
service acquisitions and assess what performance measurements are being used to gauge 
the contractor’s effectiveness. This research project will provide a literary review of 
Performance Based Contracts (PBCs), Performance Based Service Contracts (PBSC), 
Perfonnance Based Service Acquisition (PBSA) and related topics. This review will be 
conducted using various Governmental legislative and regulatory guidance and best 
industry practices. The research will look at incentives and remedies use to motivate a 
contractor to succeed performance expectations. The research will include a gathering of 
survey infonnation from contracting officers, Directors of Information Management 
offices, Information Technology (IT) technical personnel and TACOM IT contractors. 
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This information will then provide a comparative analysis for recommendations and 
conclusions. The information will then be analyzed to aid in lessons learned processes, 
guidance, and policy to aid other acquisition personnel with any performance based 
service acquisitions. To conclude, this joint applied project will include creating and 
posting performance measurement guidance for the TACOM Community website to 
provide all contracting personnel with the means to develop performance measures for IT 
performance based service contracts. 
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I. BACKGROUND 


A. INTRODUCTION 

The Department of Defense (DoD) is starting to realize that surveillance of 
service acquisitions needs to improve significantly. Even though DoD has taken steps to 
improve general oversight and management of service contracts through the National 
Defense Authorization Act for Fiscal Year 2002, these efforts have done very little to 
improve service contract surveillance (Government Accountability Office (GAO) Report 
05-274, 2005). 

Purchasing services from the private sector is not a new concept. Outsourcing, 
which is now referred to as “Competitive Sourcing”, was originally directed in the Office 
of Management and Budget (OMB) Circular No. A-76 issued in 1966. However, it is 
interesting to note that the commercial market still refers to this type of competition as 
“outsourcing.” According to this circular, the Government will not “start or carry on any 
activity to provide a commercial product or service if the product or service can be 
procured more economically from a commercial source.” Information Technology (IT) 
services have been long considered a function which private industry can perform. 
Consequently, IT services have become a target for A-76 studies a means for federal 
agencies procure services more economically thereby reducing their operational costs. 
By using competitive sourcing processes to obtain IT services, Government agencies are 
allowed to concentrate on their core mission as well reducing their overall operational 
costs. Even though the requirement to outsource services, like IT services, has been a 
requirement since 1966, it has just been in the last few years that purchases of services 
have seen a significant increase. Agencies are being pushed to use performance-based 
service contracting (PBSCs) methods, to the maximum extent possible, when developing 
service contracts (Office of Federal Procurement Policy (OFPP) Letter 91-2, Service 
Contracting and OFPP report, 2003). The use of service based contracting methods is a 
whole new arena for DOD acquisition personnel. As a result of these changes, 
surveillance of service contracts has become more important to ensure that the 
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Government agencies receive services they require in a timely and quality manner (GAO 
Report 05-0274, 2005). These developments will be discussed as part of this research. 

The purpose of this Joint Applied Project professional report is to analyze IT 
performance based service acquisitions within the U.S. Anny Tank Automotive and 
Armaments Command (TACOM) and to evaluate what performance measurements are 
being used to gauge the contractor’s effectiveness against the performance measures 
stated in the contract. Additionally, this report will also address incentives, as well as 
penalties, used to incentivize the contractor to provide the level of service envisioned by 
the Government. In conclusion, the report will analyze how development of perfonnance 
measures, incentives, and penalties affect the ability of the surveillance team to 
adequately administer the contract. 

The methodology of this research will include several steps. Literary review will 
discuss topics such as Perfonnance Based Contracts (PBCs), Performance Based Service 
Contracts (PBSCs), Performance Based Service Acquisitions (PBSAs), Quality 
Assurance Surveillance Plans (QASPs), Performance Work Statements (PWSs), and 
Statement of Objectives (SOOs), incentives, performance measurements and metrics. 
This literary review will be conducted to include government and commercial sources. 
Surveys and interviews will be perfonned and include participants from both government 
and private industry. The results of the surveys and interviews will be included in 
Chapter III. The results of the surveys and interviews will be analyzed and a comparison 
of these results, with the information from Chapter II, Literary Review, will be included 
in Chapter IV. Recommendations and areas of further research will be included in this 
research project in Chapter V. The above information will provide the PBSA Team with 
lessons learned and commercial best practices necessary to ensure successful contract 
surveillance. 

B. TERMS AND DEFINITIONS AS DEFINED IN FAR, DFARS, ETC. 

Performance-based contracting, as defined in the Federal Acquisition 
Regulation (FAR), is a means to structure all aspects of an acquisition around the purpose 
of the work to be perfonned with the contract requirements to be set forth in clear, 
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specific, and objective terms with measurable outcomes (FAR 2.101). This topic will be 
discussed further in Chapter II Literary Review. 

Performance-Based Service Acquisition (PBSA) is defined as structuring all 
aspects of an acquisition around the work to be performed with the contract requirements 
set forth in clear, specific, and objective terms with measurable outcomes rather than 
direct perfonnance processes. Simply stated, “it is a method for acquiring what is 
required and places the responsibility for how it is accomplished with the 
contractor.’’(Guidebook for Performance-Based Service Acquisition (PBSA) in the DoD, 
2001) The use of the word "acquisition" rather than "contracting" reflects the broader 
scope (from mission planning to contract performance management) and broader 
community (including program offices) that are a necessary part of the process 
(Guidebook for PBSA in the DoD, 2001). 

C. RESEARCH OBJECTIVE 

The objective of this research is to gauge the effectiveness of existing IT PBS As 
and use the lessons learned to develop and post performance measurement guidance on 
the TACOM Community website to provide all contracting personnel with a means to 
develop performance measures for IT PBSAs. 

D. RESEARCH QUESTIONS 

The research questions to be answered in this Joint Applied Project are: 

1. To what extent are performance-based measures being used to manage 
TACOM IT PBSAs? 

2. How clear and applicable are PBSA contracting policies and procedures, 
e.g., statements of work (SOWs), performance work statements (PWS), and statements of 
objectives (SOOs)? 

3. What are the incentives and penalties affecting PBSAs? 

4. What are ways to improve U.S. Army, TACOM-RI, PBSA including 
methods promulgating web-based guidance to practitioners? 
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E. SCOPE (BENEFIT TO TACOM COMMUNITY), LIMITATIONS, AND 

ASSUMPTIONS 

This research project will provide the TACOM community with 
recommendations and guidance related to developing successful performance 
measurements to use with IT Performance Based Service Acquisitions. In addition, this 
site will include lessons learned and frequently asked questions regarding the use of 
incentives (positive and negative); it will provide websites, Government documents, and 
commercial sources available to assist contracting personnel in monitoring and accessing 
contractor perfonnance; and it will provide samples of performance measures and 
incentives that have been successfully used in past and current contracts. 

This project is limited to review of IT PBSAs within the TACOM community that 
have successfully used competitive sourcing techniques or are follow-on to a competitive 
sourcing PBSA. 

Assumptions being made are that current IT PBSAs lack effective performance 
measurements, metrics, and incentives to effectively monitor of contract perfonnance. 

F. METHODOLOGY 

The methodology used for the purposes of this thesis consists of literary reviews 
of Performance Base Contracts (PBCs), Perfonnance Based Service Contracts (PBSCs), 
Perfonnance Based Service Acquisitions (PBSAs), Information Technology (IT) PBSAs, 
and surveillance of IT PBSAs to determine if adequate surveillance is cunently being 
accomplished on these types of contracts. The literary review will include Government 
Legislation, DoD Regulations, DoD policy and guidance, interim policy, and regulations; 
best practices; and best commercial practices regarding service contracts and/or 
acquisitions. 

The report will also will attempt to obtain additional insight relating to 
management of PBSAs from personnel within the TACOM community. These 
individuals are responsible for managing and monitoring IT PBSAs. This will be 
accomplished with completion of surveys or conducting interviews with Government and 
Contractor Project Managers, Directors of Information Management (DOIMs), Procuring 
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Contracting Officers, Government IT technical personnel, TACOM contractors, and other 
1102 series acquisition personnel. After receipt of the questionnaires and completion of 
the interviews, this information will be compiled and analyzed to provide information on 
what is actually happening in the TACOM community regarding surveillance of IT 
PBSAs. Based on the findings from the surveys received, this report will cover any 
interviews conducted to obtain further infonnation. 

The report will conclude with a comparison and contrast on how surveillance 
should be perfonned with what is actually being conducted. Finally, a template will be 
developed to provide guidance on the development of metrics and incentives to assist the 
PBSA Team in creation of PWSs, SOOs, QASPs, and the types of incentives necessary to 
incentivize contractor performance. 

G. BENEFIT OF RESEARCH 

Anticipated benefits of our research will include creating lessons learned and 
posting performance measurement guidance for the TACOM community, which will 
provide acquisition personnel with the means to develop performance measures to ensure 
proper surveillance of IT PBSA contracts. 

H. ORGANIZATION OF THESIS 

This report is organized into five chapters. Chapter I provided an overview of the 
research project, objectives, and methodology. Chapter II includes a literary review to 
include topics such as PBC, PBSCs, and PBSAs. Along with these topics, Chapter II will 
include definitions for the acquisition techniques and for related tasks that are required 
with these types of acquisitions. The literary review will be conducted using Government 
Legislation, regulations, DoD guidance, and commercial best practices. Chapter III will 
provide a real-world comparison based on surveys and interviews conducted from this 
research project. Additionally, Chapter III will also include a list of the questions 
contained in the survey and interviews, and the selection of the respondents and 
interviewees. Chapter IV will compare and contrast results of the literary review with the 
results of the questionnaire/interviews. Chapter IV will summarize the results of the 
analysis obtained in Chapter III. Chapter V provides overall conclusions, 
recommendations, and recommended areas for further research. 
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I. SUMMARY 

In conclusion. Chapter I of this research project states the objective of this project, 
which is to determine if IT PBSAs are being effectively measured and, if so, what lessons 
have been learned. Included in this chapter are the related terms and definitions relating 
to PBSAs. These tenns and others are further discussed in Chapter II - Literary review. 
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II. LITERARY REVIEW 


A. INTRODUCTION 

The literary that will be discussed in this chapter will include topics such as 
Performance Based Contracts (PBCs), Performance Based Service Contracts (PBSCs), 
and Performance Based Service Acquisitions (PBSAs). This will involve distinguishing 
between the PBCs, PBSCs, and PBSAs. The research will also include components and 
objectives of PBSA; therefore, additional terms will be discussed. The literary review 
will be conducted using Government legislation and regulations, DoD guidance, and 
commercial best practices. In conducting this process, the literary review will discuss 
both Government regulatory guidance and industry best practices. 

B. ELEMENTS OF PERFORMANCE BASED ACQUISITIONS 

The DoD PBSA Guidebook states that in order to be considered performance- 
based, an acquisition should contain the following elements: 

A performance work statement (PWS) describes the acquisition requirement in 
tenns of measurable outcomes rather than telling a contractor how to perform the 
supplies and/or services required. 

Measurable performance standards or metrics are used to determine whether 
desired outcomes have been met. Consequently, the standards or metrics help to define 
what will be considered acceptable performance under any resulting contract. 

Incentives and Remedies provide a means to address acceptable and 
unacceptable contractor performance. Not all acquisitions contain incentives but the team 
should be encouraged to include them whenever possible. Incentives are included to 
motivate the contractor to exceed performance standards. Additionally, this guidebook 
states “Remedies and incentives complement each other.” 

A performance Assessment plan describes how contractor performance is 

monitored and assessed. This plan can be written by the Government as part of the PWS 

or can be a requirement of the solicitation. Therefore, the contractor can include it as part 

of their proposal. If the Government writes the Perfonnance Assessment Plan, it is called 
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a Quality Assurance Surveillance Plan (QASP) while if contractor provides the 
Performance Acceptance Plan as part of their proposal, it is known as Quality Assurance 
Plan (QAP) (PBSA in the DoD, December 2000). 

Although the terms PBC, PBSC, and PBSA may mean different contracting 
methods for some people, for the purposes of this research project, the terms PBC, PBSC, 
and PBSA will be used interchangeably. Whether referred to as Performance-Based 
Service Contracting (PBSC), Performance-Based Service Acquisition (PBSA), 
Performance-Based Acquisition (PBA), or Performance-Based Contracting (PBC), the 
objective of improved performance remains the same. No longer the exception to 
traditional contracting, these perfonnance-based approaches simply make good business 
sense (KnowNet, 2005). 

The following tenns and definitions will be used during the discussion of 
contractor surveillance: 

Acceptable Quality Level (AQL) is the variance from a performance standard 
that a contractor is allowed, before the Government rejects its services. Generally 
speaking, a contractor will be given an opportunity to correct non-confonning services if 
it can be accomplished within the required delivery schedule (KnowNet, 2005). 

The Quality Assurance Evaluator (QAE) is a technical or subject matter expert 
that is qualified to perfonn the quality assurance functions required under the PBSA 
(KnowNet, 2005). 

A Contracting Officer's Representative (COR) is a government employee 
whom is appointed in writing and delegated by the Contracting Officer to perform 
specific contract-related decisions. They can also be referred to as Contracting Officer's 
Technical Representative (COTR) (KnowNet, 2005). 

C. GOVERNMENT LEGISLATION 

1. Government Performance and Results Act (GPRA) of 1993 

The rationale of GPRA was to make agencies accountable for program 
performance. GPRA was passed in response to concerns regarding Federal waste and 
inefficiencies. Like PBCs, agencies are required to establish and manage “mission- 
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related performance goals and objectives” (KnowNet, 2005). In essence, this act has laid 
the groundwork for acquisition teams to work towards result driven acquisitions instead 
of telling the contractor how to perform the requirement. Consequently, PBSAs can help 
to support the goals required by GPRA, as well as to achieve the agency’s mission. The 
PBSA team can accomplish this by clearly describing the acquisition as it relates to its 
mission, thereby setting the contractor’s goals to be in sync with that of the organization. 
(KnowNet, 2005) 

2. Clinger Cohen Act (CCA) of 1996 

The CCA states that Government IT organizations should operate like a business; 
therefore, these organizations must look for ways to streamline their processes and to run 
more efficiently. This act states the “acquisition technology planning and management” 
must be treated as a "capital investment." Therefore, organizations cannot and should not 
acquire hardware and software systems without first looking at all the facets of the 
acquisition and make a sound business decision before pursing any acquisition. 

Just like PBSA and the GPRA, the CCA requires the IT organization to “establish 
goals for improving efficiencies and effectiveness” for day-to-day operations. These 
organizations must also “ensure the established performance measurement” can produce 
the desired outcomes and are aligned with the organization’s mission. 

D. OFFICE OF FEDERAL PROCUREMENT POLICY (OFPP) 

The OFPP recognized the need to provide guidance to the acquisition workforce 
regarding PBSAs. Accordingly, the OFPP put together an interagency working group in 
April 2002 in order to evaluate and provide acquisition personnel with a better 
understanding of PBSAs. This team reviewed several internal documents which included 
OFPP “Guide to Best Practice for Performance-Based Service Contracts”, 1998, and 
OFPP Pamphlet No. 4, “A Guide to Writing and Administering Performance Statement 
of Work for Service Contracts”, 1980. In addition, the team reviewed a report prepared 
by Dr. Lawrence Martin, Pricewaterhouse Coopers Endowment for the Business of 
Government title “Making Perfonnance Based Contract Perform: What the Federal 
Government Can Learn from State and Local Governments”, which will be discussed 
later in this chapter. The task force also reviewed OFPP’s “Solicitation/Contract Task 
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Order Review Checklist, dated August 1997, and the latest revision of “The Steps to 
Performance Based Service Acquisition Guide”. 

Based on the results of this working group, a report was issued titled, 
“Performance-Based Acquisition Contracting for the Future”, dated July 2003. The 
report recommended that FAR Part 2 be revised to include definitions for performance 
work statement (PWS), quality surveillance plan (QAP), statement of objectives (SOOs), 
etc. The PWS would be used to identify the technical, functional and performance 
characteristics of an agency requirement. A QASP would be redefined to mean a plan for 
assessing the contractor’s perfonnance, which would ensure the Government’s 
performance objectives were met and were in compliance with the terms and conditions 
of the contract. 

As already defined, a SOO is alternative to the PWS. However, the team 
recommended that the SOO should provide a summary of key agency goals and/or 
outcomes. This would allow the contractor to draft, for government review, a PWS to be 
incorporated into any resulting acquisition. 

Finally, the SOW, much like the PWS, would be defined as a statement that 
describes the Government’s requirements in a clear and concise manner. 

The Interagency Working Group, chartered by the OFPP, also recommended that 
the term Performance Based Service Acquisition (PBSA) be used “to provide common 
tenninology throughout the Government”. In addition, this committee recommended that 
FAR 37.601 (a) be changed to read as follows: “The principal objective of performance- 
based service acquisition (PBSAs) is to express government needs in terms of required 
performance objectives, rather than the method of perfonnance, to encourage industry- 
driven, competitive solutions.” (OFPP Interagency Working Group, 2004) 

As result of this task force, OFPP issued memorandum Subject: “Increasing the 
Use of Performance Based Service Acquisitions”, dated September 4, 2004, and 
rescinded OFPP’s “1998 Guide to Best Practices for Performanced-Based Service 
contracting”. This memorandum stated that agencies should use the “Seven Steps to 
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Performance-Based Service Acquisition Guide, available at www.acqnet.gov when 
developing PBSAs. (OFPP Interagency Working Group, 2004) 

E. FEDERAL ACQUISITION REGULATION (FAR) 

The FAR provides a definition for PBC; however, it fails to properly define the 
differences between PBC and PBSA, as well as to include definitions of other PBSA 
terms. 

As stated in Chapter One, FAR Part 2 defines “Performance-based contracting” as 
the means of structuring all aspects of an acquisition around the purpose of the work to be 
performed with the contract requirements set forth in clear, specific, and objective terms 
with measurable outcomes as opposed to how the work is to be performed or by broad 
and imprecise statements of work. (FAR 2.101) 

Additionally, FAR 37.101, Definitions, defines a service contract as “a contract 
that directly engages the time and effort of a contractor whose primary purpose is to 
perform an identifiable task rather than to furnish an end item of supply.” (FAR 37.101) 

The policy at FAR 37.102 states that Performance Based contracts are the 
preferred method of contracting for services. Moreover, these contracts should be 
awarded under a firm fixed price whenever feasible before another type of contract is 
used. Consequently, there seems to be a need to provide and/or clarify what the 
difference is between PBCs and PBSAs. Whereas a PBC can be used to acquire 
logistical support to include both hardware and services, such as provided in a contract 
for total contractor logistical support or supply chain management, PBSAs are needed to 
provide services such as base operation support, guard services, environmental 
remediation actions, janitorial, and IT support services. 

Additionally, for any PBSA to be successful the team must adhere to FAR 
guidance which states the team must be established from “...beginning with the customer 
and ending with the contractor of the product or service. By identifying the team 
members in this manner, teamwork, unity of purpose, and open communication among 
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the members of the Team in sharing the vision and achieving the goal of the System are 
encouraged. Individual team members will participate in the acquisition process at the 
appropriate time.” (FAR 1.102-3) 

F. DEFENSE FEDERAL ACQUISITION REGULATIONS (DFARS) 

A review of the DFARs did not provide definitions for performance based 
contracts nor did it provide significant information to assist the team with the formulation 
and surveillance of any service acquisitions. 

Surveillance is required by DFARS for all PBSAs; however, DFARS does not 
specify how surveillance will be performed. Depending on the contract, organization 
surveillance ranges from formal written assessments, which are conducted monthly, 
quarterly, semi-annually, or annually, to random inspection of work orders, tickets, etc. 

G. DEFENSE ACQUISITION GUIDEBOOK (DAG) 

The Defense Acquisition Guidebook (DAG) was created to provide the 
acquisition workforce with guidance regarding the DoD Instruction 5000.1 and 5000.2. 
The DAG contains eleven (11) chapters; chapter 2 contains information regarding 
acquisition program goals. Additionally, chapter seven (7) contains information which 
informs the team of the statutory and regulatory requirements for acquiring IT. This 
chapter also provides information regarding the Clinger-Cohen Act, and it relates to the 
IT acquisitions. Consequently, this guidebook can provide the team and the acquisition 
personnel with reference guidance to enable them to develop their acquisition strategy 
(DAG, 2004). 

H. DOD DIRECTIVE 5000 SERIES GUIDANCE 

While the DoD Directive 5000 does not specifically provide information on 
performance based service contracts, it does provide the team with the conceptual 
framework for development of the acquisition. This framework indicates that, as with 
any acquisition, the team should be comprised of technical personnel, acquisition 
personnel and contractor personnel, depending on the acquisition. 
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The PBS A team should be set up from the beginning and all team members 
should be part of the market research, acquisition strategy and remain with the team 
throughout the life of the contract. 

1. DoD Instruction 5000.2 

A review of the DoD Directive 5000.2 indicates that Integrated Product Teams 
(IPTs) shall be used to communicate with the stakeholders throughout the acquisition 
process. It further states that the Program Manager has the ultimate responsibility of 
coordinating these efforts (DoDI 5000.2, 2004). 

Additionally, this directive also states that the Perfonnance Based Acquisitions 
(PBAs) shall be used to the maximum extent possible. The intent is “To maximize 
competition, innovation, and interoperability, and to enable greater flexibility in 
capitalizing on commercial technologies to reduce costs, acquisition managers shall 
consider and use performance-based strategies for acquiring and sustaining products and 
services whenever feasible.” Therefore, the acquisition requirements “shall be stated in 
performance terms.” Thus, it encourages the team to constantly seek to the use of 
performance based strategies whenever possible (DoDI 5000.2, 2004 Enclosure 1). 

I. GUIDELINES - SEVEN STEPS TO PERFORMANCE BASED SERVICE 

ACQUISITION 

This web-based guide describes performance based service acquisition using 
seven steps. The seven steps are described below: 

1. Establish the Team 

An Integrated or Perfonnance Based Service Acquisition (PBSA) Team must be 
established. This team should assist in the acquisition planning, the solicitation 
development, evaluation, and award phase of the acquisition. After contract award, the 
PBSA team will be closely involved in the surveillance of the resulting contractor’s 
performance. 

2. Describe the Problem 

This step requires a description of the problem that needs to be solved. GPRA 
plays a part in this step as it requires the agency to establish and “manage” to mission- 
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related performance goals and objectives. Additionally, the team can describe the 
problem is terms of a PWS or a SOO. 

3. Examine Private-Sector & Public Sector Solutions 

Under this step, the team should examine both government and private sector 
solutions. This is otherwise kn own as Market Research. Market research is a good 
source for finding the appropriate contract type to fit a specific situation. It is up to the 
PBSC team to make a good business decision and utilize negotiations with the contractor 
to determine the right contract type. 

In order to make a “good” business decision, the PBSC team must be clear of the 
pros and cons associated with each type of contract and what type of incentives and 
disincentives they provide. 

A fixed price contract requires the contractor to be responsible for all costs, 
whether the contractor realizes a profit or loss under the contract is determined by his 
performance. 

A fixed price incentive contract allows for the final contract price and profit to be 
calculated based on a formula stated in the contract that relates the final negotiated costs 
to the target cost. 

A fixed price contract with award fee is used when contract perfonnance cannot 
be measured objectively. This is a way to motivate the contractor when other incentives 
are not appropriate. 

Cost-reimbursement incentive contracts are used when fixed-price contracts are 
not appropriate because there is uncertainty regarding probable costs. These types of 
contracts can be cost plus incentive fee or cost plus award fee. 

A cost-plus fixed fee contract provides the contractor with minimal responsibility 
to control costs. The contractor is reimbursed for all allowable and allocable costs and 
their profit (fee) is negotiated and fixed. 

The current regulatory guidance provides a preference to utilize firm fixed price 
contracts for IT PBSAs. In the past, contracts that provide for support services have been 
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awarded as cost reimbursement type contracts, and the contractor community is very 
reluctant to change from reimbursement contracts to firm fixed price contracts. 

The argument for firm fixed price contracts for IT PBS As is these types of 
contracts state desired outcomes and identifies how performance will be measured. In a 
contract where performance is specified, but price is constrained, it is possible that 
performance might exceed expectations. 

Please note the contract does not have to be one of the contract types listed above; 
it can be a hybrid contract that contains both fixed price and cost reimbursement 
performance measures. 

4. Develop PWS or SOO 

Step 4 discusses the development of a PWS or SOO. While a PWS provides the 
contractor with desired results and measurable, relevant standards, a SOO provides a 
prospective contractor with the desired objectives. A contractor then submits a proposal 
in the form of a PWS for the Government to review. As KnowNet states, “a SOO is an 
alternative approach to using a PWS.” A SOO may ensure the contractor can meet or 
exceed the desired contract perfonnance. When drafting a SOO for an IT PBSA, the 
following objectives can be employed: 

Ensure that customer support is available 24/7 

Ensure that email services are operational 24/7 

This allows for flexibility of the contractor’s workforce to cover emergency and 
after-hour requirements. 

5. Decide How to Measure & Manage Performance 

According to this guide, “this step establishes the strategy of managing the 

contract to achieve planned perfonnance objectives. The OFPP’s “A Guide to Best 

Practice Performance Based Service Contracting”, dated October 1998, states that a QAP 

defines what the Government must do to ensure that the contractor performs to the PWS. 

Moreover, a QAP or QASP is a plan for which the Government QAE measures the 

contractor’s perfonnance against the standards contained in the PWS. This plan should 

be clear, concise, and inform the contractor what surveillance methods will be employed 
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in any resulting contract. The QASP should focus on the critical tasks contained in the 
PWS, which are measurable and attainable. It tells the contractor how they will be paid 
for services rendered to ensure the Government receives the services required. 

Consequently, the QASP provides the Government with an indication of what 
resources are needed to properly administer and assess the contractor’s performance. 
Even so, the contractor should not rely solely on the QASP to tell them how to perform 
any required services. 

In developing the QASP, the Government team should look at the PWS and 
decide which tasks are critical and worth measuring. It is better to choose fewer metrics 
which are meaningful and relevant than to select numerous or complex metrics. When 
selecting the metrics, the team should look at the metrics to ensure they are easy to 
collect. If not, they may not be achievable and are not worth being selected as a metric. 
To be achievable means the contractor will not require any assistance from Government 
personnel to provide the required services. If the contractor requires Government 
assistance, why assess or include this metric since it requires both parties to achieve the 
metric. Monitoring the contractor’s performance can be costly, especially if the 
Government chooses to monitor the contractor for 100% compliance. The question that 
should be asked before you require 100% compliance is will this actually be required to 
get the level of performance we need. We think if the “right contractor” is chosen, 100% 
compliance will not be necessary. 

As stated in Step 6 of the 7 steps to Perfonnance Based Service Acquisitions, the 
“right contractor” must be chosen and the Government should build a working 
relationship with the contractor in order for both to have a true understanding of the work 
required under the contract. 

Additionally, step 5 includes development of the contractual language, 
establishment of the contract type, incentives, metrics, etc. 

6. Selecting the Right Contractor 

Step 6 involves the developing an acquisition strategy, which will assist the team 
in selecting a contractor who provides the best value for a given service. The best value 
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contractor may not provide the lowest price, but may offer a better approach to the 
requirement or may offer a lower risk to the Government due to a higher past 
performance rating. When evaluating each contractor, the PBSA team must ensure each 
offeror understands PBSAs and has the experience/knowledge/expertise to perform the 
services required by the solicitation. 

7. Manage the Contractor’s Performance 

This step is where “the real work” of the acquisition comes into play. The agency 
must ensure that the proper resources are allocated to ensure the contractor performs 
IAW PWS. The Contracting Officer must inform the team of their responsibilities and 
limitations during this phase to avoid any unauthorized actions. This step, contract 
performance “is guided far less by law, regulation, and policy than those described in the 
preceding steps.” Consequently, the team must be aware of the terms and conditions of 
the contract and the PWS to ensure they are acting within their authority (Seven Steps to 
PBSA, 2005). 

J. GENERAL ACCOUNTABILITY OFFICE (GAO) 

The Department of Defense (DoD) has reported that DoD agencies need to 
improve surveillance on their service contracts. In the GAO report titled, “Contract 
Management, Opportunities to Improve Surveillance on Department of Defense Service 
Contracts”(March 2005),” GAO states that the problem with surveillance revolves around 
improperly trained service personnel, surveillance personnel not being assigned to the 
service contract by the date of contract award, surveillance personnel not being held 
accountable for their duties, incomplete documentation of surveillance personnel, and 
contract surveillance taking a back seat to awarding contracts. Additionally, the GAO 
report states that these types of contracts lack the sufficient reporting requirements to 
ensure that services are required and warranted. The report also states that without 
proper documentation and surveillance, DoD is at risk of being unable to identify or 
correct poor performance in a timely manner. In conclusion, GAO recommended that 
DoD require properly trained surveillance personnel be assigned at the onset of the 
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contract award. In addition, GAO stated that the surveillance personnel must be held 
accountable regarding their duties and responsibilities for each service contract (GAO 
Report 05-274, 2005). 

GAO also recognized the need for DoD to revise current PBS guidance on the 
proper use of other agencies’ contracts, to include guidance on conducting surveillance of 
services that are procured from other agencies. Finally, GAO directed that the Anny 
assign surveillance personnel to conduct surveillance of ongoing Contract Advisory and 
Assistance Services as a means of ensuring that service contracts are appropriately 
identified and monitored. This should ensure that the Army is procuring services that are 
valid and required (GAO Report 05-274, 2005). 

With GAO report 02-1049 titled, “Guidance Needed for Using Performance 
Based Service Contracting”, dated September 2002, GAO has raised the concern that 
agencies did not have a clear understanding of PBSAs; therefore, agencies were not 
taking full advantages of these types of contractual instruments. In this report, GAO sited 
that the Office of Federal Procurement Policy (OFPP) was aware of this problem and was 
working on developing new guidance on the use of perfonnance-based contracting (GAO 
Report 02-1049, 2002). 

Consequently, this report recommended that OFPP provide a revision to their 
PBS A guidance to ensure that PBSAs were being used appropriately and that proper 
government oversight was being used (GAO Report 02-1049, 2002). 

K. COMMERCIAL BEST PRACTICES 

1. Metrics 

In his article, “Performance-Based Service Metrics in IT,” Peter S. Adam states 
that for proper surveillance of IT PBSCs, metrics are the key (Adam, 2003). 

It is critical for perfonnance standards to be directly li nk ed to the desired 
outcomes. In addition, the performance standards have to be measurable, relevant, 
controllable, and achievable (Adam, 2003). 
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In order to be measurable, standards must be quantifiable and objective. In the 
case of IT PBS As, the measurements must be subjective, so a customer satisfaction 
survey may be the best way to measure/assess contract performance (Adam, 2003). 

To be achievable, only a few meaningful metrics should be used, and they must 
be set at acceptable service levels. Five to eight metrics are considered adequate and 
must be clearly communicated to all stakeholders (Adam, 2003). 

Relevant metrics must be limited to only those metrics that are truly important to 
program objectives. When developing metrics, you must also be sensitive to the cost 
associated with measuring each objective to ensure the cost associated with measuring 
the objective does not exceed the value of the measurement. Expensive metrics should 
be limited to only those requirements that are mission- critical (Adam, 2003). 

Assigned metrics must be controllable by the contractor. If there are too many 
variables associated with the metrics, the contractor may not be responsible for all the 
variables that affect the metric (Adam, 2003). 

2. Proper Alignment 

Aligning the performance metrics with end user objectives and customer needs 
can be very difficult. Metrics can be either position based or component based (Adam, 
2003). 

Proper alignment of metrics can be very difficult as the end-users, contractors, 
and the government may have different perspectives for the performance of IT contracts 
(Adam, 2003). 

What the end-users want can be expressed as availability, performance, and 
affordability. Availability may be computer systems available for the customer’s use 
when and where they are needed. Whereas performance may require computer systems to 
work as expected without interruptions and slowdowns (Adam, 2003). 

Finally, you must determine if the proposed metrics are affordable and how they 
affect the “complete” cost of the system. When negotiating the contract, the parties 
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involved must consider trade-offs between price and perfonnance, resulting in a contract 
that meets the end user’s need at an affordable price (Adam, 2003). 

3. Customer Need/Contract Specifications 

Alignment of customer needs and contract specifications are a must. To ensure 
proper alignment, it can be beneficial to have the contractor propose, or at least contribute 
to, the metrics and the quality assurance plan (QAP). This allows contractors the 
opportunity to develop their own solutions to the requirements of the acquisition. If the 
contracting office develops the metrics and QAP on their own, they could be limiting 
what the contractors are allowed to propose. The QAP must reflect reform-mandated 
changes in quality control and assurance of contracting practices (Adam, 2003). 

With the Government moving away from selecting the lowest cost, technically 
acceptable proposal, and moving towards selecting proposals based on best value 
techniques, more emphasis is placed on quality management and past-perfonnance 
evaluation, which requires totally different incentives (Adam, 2003). 

4. The Quality Approach 

In accordance with The Air Force Instruction 63-124 (April 1, 1999), when 
utilizing the quality approach, contracts are tailored to manage risk and costs; best value 
source selection criteria is used that reduces oversight and ensures that only services 
requested are received and paid for; relies on customer feedback to validate contract 
conformance or nonconformance; allows flexibility in oversight to ensure consistency, 
even under changing circumstances; allows contractors, as part of their quality assurance 
system, to report and perfonn on surveillance of services; and confirms surveillance 
results with oversight provided by the Government Quality Assurance (QA) or a third 
party audit (Adam, 2003). 

The PBSA team must realize that contract award does not mean that performance 
metrics are set for the life of the contract; they must continually be reviewed and revised. 
Therefore, the contract must contain procedures for changing metrics and measures as 
situations arise (Adam, 2003). 
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5. 


Incentives 


Incentives are vital when acquiring PBSAs, as well as other types of contracts. 
Contract incentives can be monetary or non-monetary, positive or negative. The goal of 
an incentive is to motivate and encourage the contractor to provide the best quality 
performance (Adams 2003). 

The choice of the type of contract utilized is a non-monetary incentive and, when 
utilized, can be an incentive in an IT PBSA. A contractor’s performance can be tied to 
delivery and performance incentives using speed or responsiveness as an indicator of 
meeting the metric. Incentives can also be based on meeting performance standard 
targets, not just meeting minimum contract requirements (Adam, 2003). 

Extension or reduction of a contract for good or sub- par performance can also be 
used as an incentive in the contract. If the contractor fails to perform as required, the 
contract ends at the end of the base period (Adam, 2003). 

The contract must clearly state the consequences of the levels of performance in 
terms such as minimum, maximum, and estimated, and the office responsible for 
administering the contract must be prepared to provide the appropriate consequences 
(Adam, 2003). 

The incentive structure must be geared to the acquisition, the market place, and 
the objectives to be achieved; there is no “one-size-fits all” solution (Adam, 2003). 

Performance incentives can be negotiated and should recognize the private 
sector’s most important motivator - profit. The contract should be developed so that the 
Government, as well as the contractor, can benefit from economies of scale, efficiencies, 
and new innovations. Utilizing correct incentives and ensuring the customer’s needs are 
met will minimize risks and ensure effective performance (Adam, 2003). 

Cost based incentives are performance incentives that are designed to tie profit or 
fee to the results achieved by the contractor in relation to cost based targets. An example 
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for an IT PBSA would be system availability rates. No matter which type of measure 
you choose, the perfonnance measure must be quantified and within a reasonable range 
(low-target-high) (Adam, 2003). 

Award-Fee contract arrangements use evaluation factors provided in the award- 
fee plan to subjectively assess a contractor’s performance for a specific evaluation period. 
Award fee pools are established at the beginning of the evaluation period and contractors 
are allowed to earn a portion of an award fee pool (or if performance allows, all of the 
pool) (Adam, 2003). 

Award-term contract arrangements provide additional periods of performance in 
lieu of monetary consideration for quality performance and are a fairly new concept. The 
contractor’s incentive is tied to their performance and if perfonnance is acceptable then 
the contractor gains an additional perfonnance period. For perfonnance that is 
substandard, the performance period can be shortened. These contracts are different from 
contract options as the award term contracts are based on formal evaluation processes, 
instead of regulatory procedures as required with priced options. In other words, the 
Government extending the contract tenn up to 10 years can be a reward good 
performance (Adam, 2003). 

Schedule Incentives - this type of incentive focuses on the contractor meeting or 
exceeding the delivery schedule. These incentives can be stated using calendar days or 
months, meeting or exceeding contract milestones, or meeting urgent contract 
requirements (Adam, 2003). 

A contractor’s past performance rating can affect their ability to continue to 
perform past the base year and can affect their ability to obtain other government 
contracts (Adam, 2003). 

In the article “Making Performance-Based Contracting (PBC) Perform: What the 
Federal Government Can Learn from State and Local Governments”, the author, 
Lawrence Martin, discusses how the Federal Government can learn from State and Local 
Governments in the way they approach PBCs. The article offers the following lessons 
learned from experiences in state and local governments and provides recommendations 
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to the Office of Federal Procurement Policy (OFPP). Five lessons learned and four 
recommendations were selected from this article and they are explored below (Martin, 
2002 ). 

6. Lessons Learned 

a. PBC at the state and local government levels defines 
“performance ” as consisting of outputs, quality, outcomes, or any combination thereof 
(Martin, 2002). 

Confusion appears to exist at the federal level in terms of what constitutes 
“performance” in PBC. “Performance” in state and local government PBCs mean 
outputs, quality, outcomes, and various combinations. This tripartite conceptualization of 
performance: (1) provides greater clarity as to the purpose of PBC, (2) gives state and 
local governments more options in structuring PBCs, and (3) aligns more closely the 
concept of performance in PBC with the concept of performance contained in the service 
efforts and accomplishments reporting initiative of the Governmental Accounting 
Standards Board (Martin, 2002). 

b. PBC at the state and local government levels involves varying 
degrees of being performance based (Martin, 2002). 

A goal of PBC is to make less use of design specifications (input and 
process) and more use of performance specifications (outputs, quality, and outcomes). 
The experience of state and local governments demonstrates that service contract 
specifications can be conceptualized on a continuum (inputs —> process —> outputs —> 
quality —> outcomes) from “non-PBC” to “full-PBC”. At the non-PBC end of the 
continuum are service contracts that make exclusive use of design specifications; at the 
full-PBC end of the range are service contracts that make exclusive use of outcome 
performance specifications. Conceptualizing PBC on a continuum allows for an 
incremental and developmental approach to be taken with certain services and with 
certain contractors (Martin, 2002). 

c. PBC at the state and local government levels challenges the 
notion that there is one best way to do performance- based contracting (Martin, 2002). 
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The state of the art in PBC today is insufficiently well developed to make 
any claims about how best to implement this new form of service contracting. What can 
be said is that state and local governments utilize a variety of different approaches in the 
implementation of PBC. The common denominator in these various approaches is the 
conscious attempt on the part of state and local governments to change the behaviors of 
the contractors to focus more on performance (Martin, 2002). 

d. PBC at the state and local government levels include share-in¬ 
shavings contracting, revenue enhancement contracting, and milestone contracting 
(Martin, 2002). 

The experience of state and local governments clearly demonstrates that 
share-in-savings contracting, revenue enhancement contracting, and milestone 
contracting can affect the behavior of contractors to focus more on perfonnance. Thus, 
these approaches warrant being called PBC. In the case of share-in-savings and revenue 
enhancement contracting, contractor behavior is changed to focus on accomplishment of 
certain processes and outputs that lead in turn to the accomplishment of certain desired 
outcomes (reduced service delivery costs and increased revenues). In the case of 
milestone contracting, contractor behavior is changed to focus more on performance 
because output, quality, and outcome performance requirements, as well as incentives 
and penalties, are automatically built into the contract (Martin, 2002). 

e. The step-up/step-down method is a useful approach to 
structuring incentives and penalties (Martin, 2002). 

Structuring incentives and penalties to step-up and step-down from the 
performance standard or acceptable quality level makes it clear to contractors the 
implications of acceptable performance. Step-up/step-down incentives and penalties tend 
to keep the contractor focused on the performance standards. Additionally, the contractor 
also has positive and negative motivations to achieve the performance standards. When 
incentives and penalties step-up/step-down in similar fashion, and in similar quantities, 
an appeal is also made to fairness (Martin 2002). 
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7. 


Recommendations 


a. Operationally define “performance” to include outputs, quality, 
outcomes, or any combination thereof (Martin, 2002). 

An operational definition of “performance” will resolve the current 
conflicting and contradictory operational definitions of PBC provided by OFPP, which 
views PBC as contracting for outputs and quality; the DoD, which views PBC as 
contracting outcomes; and the FAR, which views PBC as outcomes and quality (Martin, 
2002).By providing an operational definition of “performance”, this will ensure that 
Federal PBC takes a comprehensive view of performance to include considerations of 
outputs, quality, and outcomes (Martin, 2002). 

b. Recognize that varying degrees of performance based can exist in 
an acquisition (Martin, 2002). 

This will recognize PBC as existing on a range from “non-PBC” to “full- 
PBC” and will allow for an incremental and developmental approach to be taken with 
some services and with some contractors. The current Federal perspective is, a service 
contract is either perfonnance-based or not depending on the presence or absence of four 
essential elements: performance requirements, performance standards or acceptable 
quality levels, a quality assurance or monitoring plan, and positive and negative 
incentives, if the service is mission critical or involves a large expenditure of public funds 
(Martin, 2002). 

The “all or nothing” Federal perspective may result in federal departments 
and agencies choosing not to attempt a PBC with some services and with some contractors 
(Martin, 2002). 

c. Include share-in-savings, revenue enhancement, and milestone 
contracting as recognized optional forms of PBC (Martin, 2002). 

This will recognize the validity of existing practices. Share-in-savings, 
revenue enhancement, and milestone contracting all seek to change the behavior of 
contractors to focus more on perfonnance (outputs, quality, and outcomes). As such, 
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they constitute optional forms of PBC and should be so recognized and will help to begin 
the process of identifying and classifying various approaches to PBC (Martin, 2002). 


d. Promote the use of the step-up/step-down method for structuring 
incentives and penalties in PBCs (Martin, 2002). 

This recommendation will provide the Federal Government a simple and 
easy way for structuring incentives and penalties. By structuring incentives and 
penalties to step-up/step-down from performance standards or acceptable quality levels; 
it will make it clear to contractors the implementations of acceptable and unacceptable 
performance. When incentives and penalties are developed in this fashion, an appeal is 
made to fairness (Martin, 2002). 

In the article “A Vision of the Government as a World Class Buyer: Major 
Procurement Issues for the Coming Decade”, the authors, Professor Jacques S. Gansler 
and Roger C. Lipitz, discuss the challenges facing the government regarding the effective 
and efficient procurement of goods and services. A challenge they discuss in this article 
looks at changing the current acquisition process (Gansler, 2002). 

The government must learn to use incentives, instead of regulations as a 
way to create higher performance from their contractors and to be able to lower overall 
costs. If the contractors are rewarded for improving their performance and lowering their 
costs, they will make every effort to make this happen. Incentives provided to the 
contractors can be in the form of additional business or profit. The government should 
think in terms of “sharing the benefits” with the contractor to provide an additional 
incentive for improving performance and lowering costs. In addition, you must have 
penalties in place in the event the contractor does not improve his perfonnance and/or 
lower his costs on the contract. The contractor should never be given additional business 
when they repeatedly do not meet performance or cost measures. In the commercial 
world, “past perfonnance” weighs heavily in the selection of sources, and this should be 
used in government procurements as well (Gansler, 2002).Significant improvement in the 
government’s way of business in determining their needs will require a significant shift 
from buying goods to buying sophisticated services. When buying sophisticated services, 
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the procurements must be “performance based”. This requires the government to simply 
tell the supplier what is needed instead of thinking of it as buying labor hours at the 
lowest rate possible. If the govermnent wants a high quality service, that is what they 
must ask for and let the suppliers decide how to many people it will take and what quality 
of people will be required to do the job. The government must then learn how to manage 
and oversee the contractors. Just because the contract was given to a private firm, does 
not remove the government from the responsibility of managing and overseeing the 
contract (Gansler, 2002). 

This shift will require considerable training for the government 
workforce in learning how to buy and manage these sophisticated services and how to 
create appropriate incentives to encourage the contractors to improve the quality of their 
services and to lower costs (Gansler, 2002). 

L. SUMMARY 

In conclusion, this chapter provides the literary review of PBC, PBSC, and PBSA. 
Additionally, the chapter reviewed the elements required in any PBSA. In performing 
the literary review, the research project reviewed past and current legislative guidance 
and policies on PBC and PBSA, as well as reviewing commercial Best Practices. 

Chapter III will include survey and interview questions and their results. Chapter 
IV will compare these findings with the expected results gathered during the Literary 
Review in Chapter II. Chapter V will provide recommendations, conclusions and areas 
for further research. 


27 



THIS PAGE INTENTIONALLY LEFT BLANK 


28 



III. SURVEY AND INTERVIEW RESULTS 


A. INTRODUCTION 

In order to gauge the effectiveness of existing IT PBS As and to detennine if 
performance measurements, metrics, and incentives are working effectively in 
monitoring these contracts, surveys and interviews were conducted. 

B. PURPOSE OF SURVEY/INTERVIEWS 

The purpose of the survey is to compile cross-comparative data regarding IT 
PBSAs across Government facilities. A summary of findings will be developed after 
compilation of the data. This survey is by no means a scientific survey, does not 
represent a statistical sampling, and the results are not to be determined representative of 
results across the Department of Army. The intention of this survey was to get a feel of 
what procurement contracting staff, government technical staff, and contractors are 
experiencing with their PBSAs and to compare it with the experiences of the authors. 

C. DESCRIPTION OF THE PROGRAMS 

The program(s) perfonned by the interviewees and survey personnel included a 
range of IT services. The contractor personnel who were interviewed and/or surveyed 
perform IT services for several installations. A sample of the IT services provided a 
range from Project Management support, help desk services, electronic mail support 
services, blackberry and/or telephone support, to end user support. Additionally, these 
individuals also assist the Government’s Most Efficient Organization (MEO) at Sierra 
Army Depot with their reporting requirements. In essence, they provide an outside audit 
of the Government contract for IT services. 

The Government personnel who were interviewed and/or surveyed provide 
surveillance and management to include the Army’s vision of a single Director for 
Information at each site and/or installation. Currently at Rock Island Arsenal, there are at 
least three Directors for Information Management who provide a portion of the total 
information services for Rock Island Arsenal. 
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D. SELECTION OF RESPONDENTS 

To ensure all members of the PBSA team were allowed an opportunity to provide 
input, selected surveys were sent to Government Product Managers, Government IT 
Technical Staff, Warranted Contracting Officers, Directors of IT Management (DOIMs), 
TACOM contractors, and other Government procurement staff. Selected interviews were 
conducted with the President of Vista Corporation, a TACOM contractor, and the 
Director of IT for Rock Island Arsenal. Ten surveys were distributed and six were 
completed and returned. 

E. SURVEY QUESTIONS 

What is your role in IT contract services? 

Government Procurement Staff 

Government Technical Staff 

Contractor 

Question 1: 

Part A: What type of contract are you currently under for IT Services? 

Firm Fixed Price 

Cost Plus 

Time and Materials 

Combination of Any of the Above 

Other 

Part B: What is your opinion regarding effectiveness of this contract? (Were 
desired outcomes accomplished?) 

Part C: Do you feel this was the appropriate type of contract? Does it meet your 
anticipated objectives and outcomes? 
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Question 2: 

Part A: What performance metrics were contained in the solicitation/contract to 
monitor performance (cost/schedule/performance metrics)? 

Part B: Were you involved in the formulation of these metrics? 

Question 3: 

Part A: How often do you meet (informally or formally) with the 

Government/Contractor team to discuss performance issues? 

Daily 

Twice Weekly 

Weekly 

Twice Monthly 

Monthly 

Every Other Month 

Quarterly 

Part B: Do you feel that these meetings are productive? 

Part C: What do you believe is the primary contributing factor to the productivity 
or non-productivity of the meetings? 

Part D: If you had the ability to make one change to the meetings that would 
yield a better outcome to the meetings, what would that change be? 

Question 4: 

Part A: Are there incentives included in the contract, if so, what are they? 

Part B: Do you consider the incentives included in the contract positive or 
negative and why? 

Part C: If incentives are offered, how would you change them to make them more 
effective? 
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Question 5: 

Part A: Is there a direct correlation between the perfonnance metrics and the 
desired outcomes? 

Part B: Is there a direct connection between the perfonnance metrics and the 
performance incentives? 

Part C: Why would you add, change, or not change any of the current 
performance metrics? 

Question 6: 

Are the policies and procedures applicable and clear for development of IT 
PBS As SOW, PWS, and SOOs? 

Question 7: 

In your opinion, what is the primary and most important change that needs to be 
implemented to improve the Army’s IT PBSAs? 

A copy of the survey form is at Exhibit 1. 

F. RESULTS OF SURVEYS 

The survey results are as follows: 

What is your role in IT contract services? 

Government Procurement Staff 

Government Technical Staff 

Contractor 

One respondent was a member of the government procurement staff, four were 
government members of the technical staff, and one respondent was a TACOM 
contractor. 

Question 1: 

Part A: What type of contract are you currently under forIT Services? 
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Firm Fixed Price 


Cost Plus 

Time and Materials 

Combination of Any of the Above 

Other 

Of the six respondents, the TACOM contractor and two members of the 
government technical staff stated their IT service contracts were firm fixed price. The 
other three respondents, all members of the government technical staff, stated their IT 
service contracts were a combination of firm fixed price, cost plus, and/or time and 
materials. 

Part B: What is your opinion regarding effectiveness of this contract? (Were 
desired outcomes accomplished?) 

When asked their opinion regarding the effectiveness of their firm fixed price 
contracts, the TACOM contractor and two government technical staff members agreed 
that firm fixed price contracts were not the best solution or the most effective way to 
manage an IT service contract, especially with regards to contractor perfonnance. 

The remaining three respondents, all government technical staff members 
working with combination contracts, felt these contract types were very effective. One 
respondent stated regardless of the type of contract, the management of cost, schedule, 
and performance is influenced by the education, training, and experience of the project 
managers, Contracting Officer’s Representatives (CORs), and perfonnance certifiers. 
This respondent stated from her experience, this is typically the largest influencing factor 
related to the effectiveness of the contract, not the contract type. 

Part C: Do you feel this was the appropriate type of contract? Does it meet 
you anticipated objectives and outcomes? 

When asked if they felt the contract type was appropriate, the respondents, the 

TACOM contractor and two members of the government technical staff who had used 

firm fixed price contracts, felt this type of contract did not provide an incentive for 
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contractors to continue providing better solutions to problems and they also did not allow 
for incorporating changes in technology and to anticipate future workload requirements. 
In addition, the respondents felt firm fixed price contracts were a challenge to administer 
and to perform as well. 

The remaining three members of the government technical staff felt their 
combination contracts allowed them flexibility to work with the contractor in the ever- 
changing IT arena. 

Question 2: 

Part A: What performance metrics were contained in the 

solicitation/contract to monitor performance (cost/schedule/performance metrics)? 

The respondents were asked to describe the perfonnance metrics contained in 
their solicitation/contract to monitor performance. 

None of the respondents actually specified the performance metrics that were 
included in their solicitation/contract. Five of the respondents, the TACOM contractor, 
three members of the government technical staff, and one government procurement staff 
member stated performance metrics contained in their contracts primarily focused on 
schedule and performance; these included successful implementation of applications, 
reporting requirements, a detailed perfonnance requirements summary, and remedies for 
sub-standard perfonnance. The remaining respondent, a member of the government 
technical staff, stated their contract contained varied metrics and QASP requirements 
which ranged from successful implementation of applications to reporting requirements. 

Part B: Were you involved in the formulation of these metrics? 

In order to detennine who is being included in the development/formulation of the 
metrics placed in TACOM IT PBSAs, the respondents were asked if they were involved 
in the formulation of the metrics in their contracts. Two individuals from the government 
technical staff and the one individual who is part of the government procurement staff 
stated they were involved in the formulation of the contract metrics. The other two 
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individuals who are part of the government technical staff and the TACOM contractor 
stated they were not part of the fonnulation of the metrics for their contract. 

Question 3: 

Part A: How often do you meet (informally or formally) with the 
Government and/or the Contractor team to discuss performance issues? 

Daily 

Twice Weekly 

Weekly 

Twice Monthly 

Monthly 

Every Other Month 

Quarterly 

In an effort to determine the level of surveillance being done on IT PBS As, 
respondents were asked how often the government/contractor team meets to discuss 
performance issues. 

Two respondents, both government technical staff members, stated their teams 
have daily meetings; a procurement staff member stated their team meets twice a week; 
one government technical staff member stated their team meets weekly, monthly, and 
quarterly; another government technical staff member stated their team meets monthly; 
and finally, the TACOM contractor stated he meets with the government team every 
other month. 

Part B: Do you feel that these meetings are productive? 

When asked if their meetings were productive, the TACOM contractor, the 
government procurement staff member, and three government technical staff members 
stated their meetings were productive, while one government technical staff member 
stated their meetings were not productive. 
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Part C: What do you believe is the primary contributing factor to the 
productivity or non-productivity of the meetings? 

In order to determine why meetings were or were not productive, the respondents 
were asked what they believed was the primary contributing factor that made the meeting 
productive or nonproductive. 

One member of the government technical staff stated having an agenda, a 
structured format for the meeting, and clear/concise expectations made their meeting 
productive. In addition, this respondent feels communication must be simple and frank 
and that talking around issues is ineffective in getting them resolved. 

Another government technical staff member feels open discussions with the 
contractor and understanding their concerns will help the team stop a problem before it 
happens. Limiting discussions to critical issues, repetitive performance problems, and 
high cost areas are contributing factors stated by one government procurement staff 
member and a government technical staff member would help make meetings more 
productive. 

A contractor who is unwilling or unable to perform is stated as a contributing 
factor by one of the government technical staff members. Finally, the respondent, who is 
a TACOM contractor, feels meetings are necessary due to a difference in understanding 
of issues by the government and industry. He feels these meetings foster an 
understanding of the contractor’s perfonnance and help build trust between the customer 
and the contractor. 

Part D: If you had the ability to make one change to the meetings that would 
yield a better outcome to the meetings, what would that change be? 

When asked if you had the ability to make one change in the meetings in order to 
yield a better outcome, what would that suggestion be? The following responses were 
received. 

Provide a clear and concise expectation with clear measurable outcomes was the 
suggestion from one of the government technical staff members. 
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Another government technical staff member stated he would limit the acceptable 
range of allowable excuses for failed perfonnance to reasonable issues only. If either 
party failed due to their lack of appropriate application of resources, accept the deficiency 
and move on. 

All parties should be less adversarial states another government technical staff 
member. 

The government procurement staff member feels limiting discussions to the most 
critical issues would increase productivity during their meetings. 

During the meetings, suggest an improvement to the perfonnance/reports so they 
can be brought to the next meeting was a suggestion by the TACOM contractor. 

Question 4: 

Part A: Are there incentives included in the contract, if so, what are they? 

Part B: Do you consider the incentives included in the contract positive or 
negative and why? 

To determine what incentives are being used in TACOM contracts, each 
respondent was asked to describe what incentives were used in their contracts and if the 
incentives were positive or negative. 

Two respondents, one a government technical staff member and the other a 
TACOM contractor, stated there were no incentives in their contract. Though, the 
government technical staff member went on to say the government, in most cases, isn’t 
comfortable or familiar with trying to include financial incentives, at least from his 
experience. Other incentives are typically not used or encouraged - it takes consistent 
management of the contract to ensure performance and validation in order to approve 
incentives, which isn’t always done or available through the government team. 

Another respondent, a government technical staff member, stated there were no 
positive incentives in their contract, but there were a significant amount of penalties for 
non-perfonnance of the contract. 
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Two respondents, a government procurement staff member and a government 
technical staff member, stated the incentive included in their contracts tied extension of 
the life of the contract with exceptional performance. 

The final respondent, a government technical staff member, stated incentives in 
their contract are tied to perfonnance and they are positive incentives as the contractor is 
compensated for completed work within the quality standards. If the contractor’s work is 
not in accordance with the standards, then a deduction can be made. 

Part C: If incentives are offered, how would you change them to make them 
more effective? 

When offered the chance to change the contract incentives to make them more 
effective, only two government technical staff members provided responses. One of the 
government technical staff members stated he would like to see some of the 
government’s financial regulations and the way the government handles their money 
changed. The other technical staff member stated he would make sure the government 
and the contractor fully understand the linkage between performance, compensation, 
deductions, and how they will be applied during the course of the contract. 

Question 5: 

Part A: Is there a direct correlation between the performance metrics and 
the desired outcomes? 

All respondents felt there is a direct correlation between the performance metrics, 
the desired outcomes, and how desired outcomes are measured against the contract 
provisions. The TACOM contractor stated the perfonnance metrics in their current 
contract were very ineffective in determining the performance of the contract. Without 
the valuable input of the industry representative, the chance of accurate measurements 
has been skewed. 
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Part B: Is there a direct connection between the performance metrics and the 
performance incentives? 

When asked if there is a direct connection between the perfonnance metrics and 
the perfonnance incentives, the government procurement staff member and two 
government technical staff members said yes there was; the TACOM contractor and two 
government technical staff members stated no there was not; and one government 
technical staff member did not answer this question. 

Part C: Why would you add, change, or not change any of the current 
performance metrics? 

The respondents were asked why would you change, not change, or add 
performance metrics to the current contract. 

The government procurement staff member and two government technical staff 
members did not answer this question; one government technical staff member stated 
every contract has to be considered individually to determine what needs to be changed 
and adapted based on lessons learned; and the TACOM contractor stated current metrics 
should be changed to reflect a positive direction and to provide cost effectiveness and 
performance solutions. Metrics should be present as a dashboard for improvement and 
not a chopping block if they are not met. 

Question 6: 

Are the policies and procedures applicable and clear for development of IT 
PBS As SOW, PWS, and SOOs? 

This question asks the respondents if current policy and procedures are applicable 
and clear for development of IT PBS As, SOWs, PWS, and SOOs. One government 
technical representative answered the policies and procedures were not applicable and 
clear causing the government technical staff to rely on the contracting officers for 
assistance. 

Another government technical representative answered no, but provided no 
reasons for his answer. 
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The positive responses came from the government procurement staff member, 
two government technical staff members, and the TACOM contractor. Only the TACOM 
contractor provided comments. 

The TACOM contractor stated policies and procedures are applicable, but remain 
vague in the development of IT PBSAs, SOWs, PWS, and SOOs. He feels this is good as 
each area can define their own needs and requirements for performance of IT services 
and support. The value of having an outside industry expert in developing these criteria 
is a vital step, without a common ground of understanding; often performance metrics 
will be far from the intended target. 

Question 7: 

In your opinion, what is the primary and most important change that needs 
to be implemented to improve the Army’s IT PBSAs? 

Each respondent was asked “what is the primary and most important change that 
needs to be implemented to improve the Anny’s IT PBSAs”. The answers are as follows: 

One government technical staff member stated that less emphasis should be 
placed on cost and more emphasis should be placed on other factors such as technical 
merits of the bid and the contractor’s past perfonnance. Past perfonnance needs to be 
scrutinized. 

When contracts are being awarded for IT work, one government technical staff 
member would like them to be routed through the IT office to verify the application and 
software that can be applied to equipment within the government. Sometimes unforeseen 
costs/time occurs due to lack of coordination. 

The government procurement staff member feels that more samples and lessons 
learned need to be made available. 

Develop education and consistent practices, checklists for buyers to look for as it 
relates to IT acquisitions, and increase education for the workforce on IT acquisition 
management is the suggestion of one government technical staff member. 
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One government technical staff member feels that IT personnel need more than 
classroom training to be effective in development of IT contract vehicles. Hands-on 
experience is by far the most beneficial training method. For the most part, the 
Information Management Specialist is poorly equipped to absorb and understand the 
fundamentals of the acquisition community, let alone its subtleties. 

Finally, the TACOM contractor feels that the Army should continue to look out to 
the corporate world and develop some of the metrics and incentives used in IT PBSAs. 
Dashboards, milestones, balance scorecards, and goals and controls are just some of the 
performance criteria currently being used to measure and encourage performance. The 
one thing all of these programs have in common is the customer and the contractor come 
together to discuss and detennine what the measurements should be to foster a win-win 
situation during the contract. 

G. INTERVIEW QUESTIONS AND RESULTS 

The results from both Government and Contractor IT Management Personnel are 
as follows: 

Question 1: This question asked if the type of contract was appropriate for 
the type of work required by the contract. 

Both Government and Contractor personnel interviewed are currently using firm 
fixed price contracts and felt that this type of contract was not appropriate for the work 
required under their contracts. There are areas where a fixed price line item is 
appropriate, but there are also line items that need more flexibility than a firm fixed price 
contract allows, i.e., a line item where the service involves changing technologies or 
application development/maintenance may fit better as level of effort line item. 
Consequently, the most appropriate contract type would be one that provides the most 
flexibility and can accommodate changing technology and fluctuations in budget and 
workload. 

Question 2: Did The Team Receive Any Training On PBSA? 

When asked if the team received training on PBS, the Government IT Manager at 
TACOM-RI, stated that his team did not received training on PBS. This person also 
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stated that team training was not emphasized, nor was the team involved in the 
development of the acquisition strategy or the actual solicitation/contract since the 
acquisition was developed under the A-76 program. The secrecy of the A-76 process did 
not lend itself to ensuring that all parties had a thorough understanding of what type of 
training and contractual requirements were needed to ensure that the Government was 
getting a fair comparison from both the Government and private industry. 

While the government personnel did not receive training, both TACOM IT 
contractors stated that their firms ensure that each program manager is provided training 
on PBSA. In fact, one of the TACOM IT contractors stated that his program managers 
receive the same training that is offered to Government acquisition personnel. This 
contractor feels that this helps his company in building a relationship with the 
Government after the contract is awarded. In his opinion, he feels that it enables his 
company to help the Government further define the desired outcomes as the contract is 
performed and to deal with the unintended consequences as they occur. 

Question 3: Were the desired outcomes clearly stated in the contract? 

Both the Government and Contractor interviewees stated that the desired 

outcomes were not clearly stated. One of the TACOM IT Contractors stated that he 

would review sampling procedures/requirements in the QASP and make sure that 

processes were in place to insure adequate surveillance of the contract. However, the 

Government manager did state that there was no distinction between the critical and 

routine components of the services required under the contract and, in some cases; this 

diluted the effectiveness of having the QASP. Two of the three interviewees stated that 

the QASP is not effective in achieving the desired outcomes because the Government 

failed to properly identify the desired outcomes from the onset. As the Government 

Manager stated in his interview, the unintended consequence of not clearly defining the 

desired outcomes or not knowing what truly was a desired outcome resulted in the 

Government being forced to perform more detail surveillance than originally intended. 

Consequently, this was a very costly consequence of outsourcing for both the 

Government and the private contractor. The private contractor agreed that the 

Government did not clearly define their desired outcomes. He stated that the 
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Government does not necessarily know how to define their metrics. In his opinion, the 
Government personnel do not know how to measure perfonnance using metrics; 
therefore, it is hard for them to know how or what their desired outcomes are. The 
contractor substantiated his statement by stating that his firm has been assisting the 
Government Most Efficient Organization (MEO), at a subordinate command, with 
refining their metrics and providing reporting data. In fact, this information provided 
with the contractor’s assistance resulted in the Government organization requiring 
additional personnel to perfonn the required services. 

In conclusion, both all three interviewees provided candid responses and stated 
that PBSA is the correct tool to use for IT PBSAs, however, there are a lot of lessons 
learned that need to be considered when the next contract is developed. 

H. SUMMARY 

Ten surveys were distributed, with six surveys completed and returned. 
Additionally, three interviews were conducted based on information received from the 
surveys. Survey and interview results have been compiled in Chapter III. Chapter IV 
compared the real world findings with the expected results gathered during the Literary 
Review in Chapter II. 
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IV. FINDINGS AND RESULTS (LITERARY REVIEW AND 
SURVEY/INTERVIEW RESULTS 

A. INTRODUCTION 

This chapter will provide a comparison of the findings discussed in Chapter III 
with the expected results discussed in Chapter II of this research project. As stated in 
Chapter II, the literary review provided the basis and rationale why PBSAs are so 
important. The importance of any PBSA is to assist the organization(s) with perfonnance 
measurements and objectives. Additionally, the surveys and interviews included in 
Chapter III indicate real world experiences. Based on these findings and results, Chapter 
IV will provide recommendations and conclusions. 

B. COMPARISON OF FINDINGS TO EXPECTED RESULTS 

In performing a comparison of the information gathered in the literature review 
with the results of the surveys and interviews, the research identified various disconnects 
or deficiencies in the PBSA process. These deficiencies and disconnects were identified 
in the following areas: establishing the PBSA team, defining the requirement, conducting 
market research, solicitation development (SOW, PWS, Measuring Performance 
Standards, Metrics, Incentives and Remedies, Awarding the Contract, and Contractor 
Surveillance. The remainder of this chapter will discuss each of these areas in detail. 

1. Establishing of the PBSA Team 

The literary review, conducted in Chapter II, stated that establishment of the 
acquisition team is essential to the success of any PBSA. To stress the importance of the 
development of the acquisition team, the FAR defines the acquisition team in the Guiding 
Principles. This is done to ensure participants are identified beginning with the customer 
and ending with the contractor who has performed the required services. By identifying 
the team members in this manner, teamwork, unity of purpose, open communication, 
team members sharing the vision and achieving the goal of the acquisition is encouraged 
(FAR Part 1.102-3). 
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The DoD Directive 5000.2 indicates that with any acquisition, the Integrated 
Product Team (IPT) should be comprised of technical personnel, acquisition personnel, 
and contractor personnel, depending on the acquisition (DoDI 5000.2, 2004). 

The results of the surveys and interviews discovered that development of the 
acquisition team was not being completed as required. One of the Government IT 
management personnel interviewed stated that, in their situation, the lack of participation 
by the PBS A team was possibly a result of the solicitation/contract being developed 
under the A-76 cost comparison process, and that the secrecy associated with this process 
did not allow the government technical personnel and contractors to work as a team. 

In addition, all survey respondents, both government and contractor personnel, 
felt that to be a more informed participant in the PBSA team, the Government technical 
personnel need to receive training in the development of IT PBSAs. Additionally, one of 
the Government IT Managers surveyed stated they felt that as part of the team they 
should also receive training in the fundamentals of acquisition. Still, another survey 
indicated that acquisition personnel needed to have a clear understanding of the expected 
results in order to assist the team with the best acquisition strategy. 

2. Defining the Requirement 

To define the requirement, the literary review states the PBSA team needs to 
describe the problem to be solved using performance specifications (outputs, quality, and 
outcomes) instead of design specifications (inputs and processes) (Martin 2002). In 
addition, when defining the requirement, the agency must develop performance goals and 
objectives, which can be included in the PWS or SOO (Seven Steps to PBSAs, 2005). 

With the exception of the government procurement staff member, only two of our 
respondents, both government technical staff members, were involved in the development 
of the requirement. 

A comment by one of the respondents was that the secrecy of the A-76 process 
did not lend itself to ensuring all parties had a thorough understanding of what type of 
contractual requirements were needed to ensure the government would get a fair 
comparison from both the government and private industry. 
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Additionally, the literary infonnation provided in Chapter II stated that industry 
members feel that the government’s inability to clearly define their requirement stems 
from a lack of training and experience in PBSAs (GAO Report 02-1049, 2002). This was 
echoed by comments made in both the surveys and interviews conducted for this research 
project. In fact, the TACOM contractor interviewed felt the QASPs are ineffective and 
should not be used in PBSAs. Consequently, the PWS and QASP are always being 
modified due to the government’s inability to clearly define requirements. 

The remaining government respondents were not included in defining the 
requirements of the acquisitions, even though they would ultimately be responsible for 
providing surveillance assistance for the resulting contract. 

However, the Government technical personnel felt the PWS clearly defined the 
requirements or at least the desired outcomes and the subsequent QASP provided the 
contractor with enough information for them to perform the required services. 
Consequently, it is important for the PBS A team to have a clear understanding of the 
services required in any PBSA (Seven Steps to PBSAs, 2005). 

3. Conducting Market Research 

Market research is the responsibility of the PBSA team. They must examine both 
public and private sector solutions to their stated requirement. Even though the 
government has stated its preference for firm fixed price contracts for IT PBSAs, the 
literary review provides that market research is a good time for determining the 
appropriate contract type to fit the stated requirement, which may be a hybrid contract 
that contains both fixed price and cost reimbursement performance measures (Seven 
Steps to PBSAs, 2005). 

The surveys and interviews did not specifically ask if PBSA team members were 
involved in the market research for their particular acquisitions. We could assume that 
the teams that chose other than firm fixed price contracts did some type of market 
research. 

In discussing this issue with the contracting officers that have worked with PBSA 
contracts at TACOM-RI, the very first PBSA contract was a result of an A-76 study. For 
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this acquisition, the PBSA team was not involved in the development of the requirement 
or in the market research. Still, the research conducted for this project did indicate that a 
sources sought synopsis was issued for this acquisition as a means of soliciting interested 
parties. As this was the only market research conducted and due to the fact the PBSA 
team was not involved, adequate market research may not have been conducted, which 
could have lead to problems with the development of the SOW, award of the contract, 
and contract surveillance. In conclusion, market research is essential to the success of any 
PBSAs (Seven Steps to PBSAs, 2005). 

4. Solicitation Development - SOW, Metrics, Incentives, etc. 

The literary review conducted in Chapter II states that the solicitation and the 
required services must be written in support of mission-based performance goals. This is 
further reiterated in Step 2 of the Seven Steps to PBSAs, wherein it states the 
performance goals set the stage for the development of the solicitation (Seven Steps to 
PBSAs, 2005, Step Two). The survey results indicated that the PBSA team did not 
necessarily know what their intended outcome or perfonnance objective was, therefore, 
they were unable to clearly identify how the mission goals related to the solicitation 
development. One of the reasons given for this problem could have been attributed to the 
secrecy of the A-76 cost comparison process and/or the lack of training and experience 
on the part of the technical personnel. 

The literary review also provided an example on how an IT PBSA could be 
structured to meet the organization’s mission. The example illustrates the need for an IT 
organization to provide a local LAN network and infrastructure that can provide its 
customers with the highest quality and service. Thus, the focus or objective of the 
acquisition should be on quality and customer satisfaction. This should enable the 
organization to meet their organizational objectives of ensuring the organization “... has 
a cost-effective, efficient, accessible, and reliable network infrastructure with modern 
workplace software and hardware, to promote productivity and meet business needs” 
(Seven Steps to PBSAs, 2005). However, as previously stated, the research surveys and 
interviews indicated that the team and the acquisition did not clearly identify the intended 
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results. As a result, the contract did not necessarily meet the needs of the organization 
and/or support the performance objectives as intended by the GPRA and illustrated in 
Figure 1 (Seven Steps to PBSAs, 2005). 

Figure 1: 

PERFORMANCE PLANNING AND PERFORMANCE- 

BASED ACQUISITION 


GPRA Strategic Plan 
Requirement 

Related Annual 
GPRA Performance 
Plan Requirement 

Incorporation in 
Acquisition 

A comprehensive mission 
statement covering the major 
functions and operations of the 
agency. 

No counterpart. 

Establishes the function to be 
supported by the technology as 
central to achievement of the 
agency’s mission, answering in 
part the first "pesky question" - 
and makes clear to contractor(s) 
the mission-related role of the 
contract. 

General goals and objectives, 
including outcome-related 
goals and objectives, for the 
major functions and operations 
of the agency. 

Performance goals, expressed 
in an objective, quantifiable, 
and measurable form, that 
define the level of performance 
to be achieved by a program 
activity. 

Enables the statement of the 
requirement to address desired 
outcomes, placing a degree of 
performance burden on the 
contractor. [Note: For all but major 
acquisitions, the GPRA 
performance goals may need to be 
expanded to detail the cost, 
schedule, and performance goals 
required by FASA.] 

A description of how the goals 
and objectives are to be 
achieved, including a 
description of the operational 
processes, skills and 
technology, and the human, 
capital, information, and other 
resources required to meet 
those goals and objectives. 

The operational processes, 
skills and technology, and the 
human, capital, information, or 
other resources required to meet 
the performance goals. [Note: 
the description in the annual 
plan may be more detailed than 
that in the strategic plan.] 

May be used to flesh out the 
statement of requirements in terms 
of an approved (by agency, OMB, 
and Congress) approach and 
required resources. 

A description of the program 
evaluations used in establishing 
or revising general goals and 
objectives, with a schedule for 
future program evaluation. 

Performance indicators to be 
used in measuring or assessing 
the relevant outputs, service 
levels, and outcomes of each 
program activity. 

The basis for comparing actual 
program results with the 
established performance goals. 

The means to be used to verify 
and validate measured values. 

Permits development of a 
complementary contractor 
evaluation process (to the extent 
that the contractor will be 
responsible for meeting 
performance goals and objectives) 
that can feed into the program 
evaluation process. [Note: For all 
but major acquisitions, the GPRA 
performance indicators may need 
to be expanded to detail how to 
measure the cost, schedule, and 
performance goals required by 
FASA.] 
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As illustrated above, the chart depicts the linkage between an organization’s 
strategic plan, their annual plan, and the PBSA. Thus, this enables the organization to 
link the funding required to meet mission objectives to performance requirements and 
desired outcomes. Note that the focus here is not on the resources required, but on what 
outcome is required. With this foundation, when the planning process is complete, an 
agency should be able to demonstrate clearly how an individual acquisition's performance 
objectives will assist in achieving the agency's mission and goals (Seven Steps to PBSA, 
2005). 

a. PWS, SOO, and/or SOW 

As the literary review indicated, once the performance goals are identified, 
the team should switch their focus to the desired outcomes and objectives surrounding 
the acquisition (KnowNet, 2005). The survey results and subsequent interviews 
indicated that the SOW, PWS, and/or the SOO did not support the desired outcomes. As 
the contractor personnel indicated in their interviews, their experience has shown that 
with IT PBS As, the SOW is often used as a living document that must be modified in 
order to get the outcomes desired. The problem with this type of philosophy is that it can 
affect the integrity of the acquisition, as other contractors did not have the same 
opportunity to submit their proposals to the revised SOW, which actually provided the 
desired outcomes. 

Additionally, the government personnel surveyed and interviewed also 
stated the PWS did not clearly identify desired outcomes. The Government technical 
team stated that they understood the intent of the IT PBSA and felt that the contractor 
also should have clearly understood what was required under this contract. However, 
their vision could have been clouded by the fact that they were the contractor’s 
competitors and had previously performed these tasks as government employees. As the 
literary review has shown, there is no standard template to provide the PBSA team with 
assistance in writing a PWS (Seven Steps to PBSA, 2005). In addition, FAR guidance 
only requires that the PWS describe the requirements in terms of results rather than 
processes: use measurable performance standards and QASPs; provide for reduction of 
fees or price; and include performance incentives as appropriate (Seven Steps to PBSA, 
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2005). For government personnel who are not familiar with PBSAs, the FAR guidance 
does not provide adequate information for development of critical parts of the PBSAs. 
This will be discussed further in this chapter and in Chapter V. 

b. Measurable Performance Standards or Metrics 

The literary review in Chapter II showed that performance standards or 
metrics play an important part in any PBS A as they are directly linked to the desired 
outcome(s) (Seven Steps to PBSA, 2005). However, the use of too many metrics can be 
costly both to the government and to the contractor. The more metrics contained in a 
contract the more technical personnel are required for contractor surveillance. 
Additionally, the contract cost increases with every metric imposed on the contractor. 
Consequently, the PBSA team should only include metrics, which are relevant to desired 
outcomes, as well as being measurable. Therefore, as a rule of thumb, PBSAs should 
only contain one performance metric for each task unless otherwise required (KnowNet, 
2005). 

In comparison, none of the survey respondents indicated that they were 
involved in the development of any metrics used on their PBSAs. In fact, the Rock 
Island Arsenal DOIM stated in that some of the requirements, under the PBSA, were 
written by another organization. Yet, the primary reason for the use of metrics in their 
contracts was to focus on schedule and performance; this included successful 
implementation of applications, reporting requirements, a detailed performance 
requirements summary, and remedies for substandard performance. Without a clear 
understanding regarding metrics being imposed under the PBSA, the technical personnel 
were reviewing all processes instead of focusing on the metrics, which could have 
assisted the organization in meeting their GPRA requirements (Seven Steps to PBSA, 
2005). The government spent additional resources gauging the contractor’s performance, 
yet it is unknown how this related to the performance standards and mission goals. On 
the other hand, both the government and contractor interviewees felt that metrics are 
important and that training was needed to assist them in this area. 
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c. 


Incentives and Remedies 


Besides ensuring the solicitation contains the appropriate metrics, the 
acquisition is also shaped by the type of contract incentives. The research showed that 
contract incentives could be monetary or non-monetary, as well as being positive or 
negative (PBSA in the DoD, 2000). It is interesting to note that the survey results and 
interviewees did indicate that their contract had negative, monetary incentives. 
Therefore, the only remedy to motivate the contractor for not meeting an Acceptable 
Quality Level (AQL) was to reduce monies based on a formula included in the contract. 
These discrepancies can be attributed to the lack of training and experience that the 
PBSA team had when they were developing the PBSA. Motivating the contractor to 
exceed Acceptable Quality Levels (AQLs) and contract expectations is a new concept for 
the DoD. The team may not have realized that they should look at the complexity, risk, 
and dollar value of the PBSA when they are formulating the type of incentive and/or 
remedies to include during the development of the PBSA. They most likely reviewed the 
FAR guidance, which states that, “Firm Fixed price contracts are generally 
appropriate...” and decided to look no further (FAR 37.602-4). Thus, the team limited 
the government’s ability to include other incentive arrangements. Consequently, PBSA 
teams generally structured their PBSAs based on firm fixed pricing. 

As one of the TACOM contractor interviewee stated that under this 
scenario, the contractor focused on perfonning the services that provided them with the 
most profit. Again, both the contractor and government personnel stated that the lack of 
training and experience could have attributed to the way the incentives were structured in 
(of) the solicitation. Another factor that was brought out during the interviews was that 
an A-76 cost comparison process appears be inflexible and does not allow a teaming 
environment and/or the sharing of ideas throughout the acquisition process. Moreover, if 
the PBSA team would have reviewed FAR 37.6 and then looked at FAR 16.103(b), they 
may have realized that they were not limited to structuring the solicitation as a firm fixed 
price contract (FAR 37.602-4). Consequently, other contract types can be considered 
“...that will appropriately tie profit to contractor performance" (FAR 16.202). 
Nevertheless, all respondents agreed that each PBSA should be structured to the meet the 
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requirements of the services required. Therefore, the PBSA team should be aware that 
there is not a "one size fits all" approach; instead, the incentive structure should be geared 
to the acquisition, the characteristics of the marketplace, and the objectives the 
government seeks to achieve (Seven Steps for PBSA, 2005). 

5. Awarding the Contract 

As discussed in Chapter II, selecting the right contractor for the job is essential in 
order to ensure that mission goals are met and customers are satisfied (Seven Steps to 
PBSA, 2005). This process can be hampered with numerous problems from improperly 
structuring the solicitation to lack of training and experience of the individuals evaluating 
offerors proposals (Seven Steps to PBSA, 2005). While the A-76 process and a protest 
can be linked to the selection of the contractor for the previous IT PBSA, it is unclear if 
the outcome would have been different if the acquisition had been structured differently. 
In interviewing the contractor personnel, they stated the A-76 cost comparison process 
tends to promote a hostile environment, no matter who is selected. Therefore, some 
companies will not even submit proposals against the government when the A-76 process 
is involved. The Government IT manager interviewed stated that with the recent A-76 
PBSA, there were numerous changes in the acquisition personnel assigned to the 
acquisition. This contradicts the intent that has been echoed in the literary review topic 
regarding the PBSA team (Seven Steps to PBSA, 2005). The government technical IT 
interviewee stated that there were five contracting officers who were involved in various 
stages of the acquisition. The research showed that the contracting officer who awarded 
the previous IT PBSA was different than the one last assigned to administer and monitor 
the contract. This situation did not help with the surveillance of the contract since the 
contracting officer administering the contract had no previous knowledge on how the 
acquisition strategy was developed. 

Still, all survey/interview respondents stated that both the IT technical and/or the 
1102 acquisition personnel lacked the expertise and training in PBSA. Without the 
expertise in PBSAs, the PBSA team had a difficult time in selecting “the right contractor” 
(Seven Steps to PBSA, 2005). Consequently, both the contractor and government 
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personnel emphasized the importance of PBS A training to ensure that the government 
can select a contractor who can provide the highest quality service at a reasonable cost 
(Seven Steps to PBSA, 2005). 

6. Contractor Surveillance 

According to the literary review, the PBSA team should also be involved in the 
contractor surveillance. Only then can the Government ensure that it is gauging the 
contractor’s performance based on the objectives stated in the PWS. As discussed in 
Chapter II, GAO recognizes surveillance is critical to the organization’s mission (GAO 
Report Number 05-274, 2005). Thus, the individuals conducting surveillance must have 
the knowledge and background necessary to gauge the contractor’s perfonnance (GAO 
Report Number 05-274, 2005). However, this assumes that the PWS has been written to 
ensure that the organization’s objectives are being met. Based on the surveys conducted 
for this project, it is unclear that the desired outcomes are being achieved. As the 
contractor interviewee stated, the Government has not clearly defined what their desired 
outcomes are and has resulted in numerous contract modifications. As one of the 
TACOM contractors stated, the government is unaware of what their desired outcomes 
are and, often, the contractor has to assist the government in determining their outcomes 
after contract award. Furthermore, this contractor stated that, in his opinion, performing 
surveillance is a new concept for the government. According to the interviewee, the 
Government has a hard time conducting surveillance since this is a new mindset in which 
the government has little or no experience. Consequently, to overcome this mindset, the 
PBSA team must ask the following questions: what are we measuring, what are the 
desired or intended outcomes, and how does this relate to the organization’s mission 
objective (KnowNet, 2005). 

Additionally, the government technical interviewee stated that they did not write 
the QASP for the previous contract; therefore, they did not know if the QASP clearly 
defined the intended results. This contradicts the literary guidance which states that the 
PBSA team should prepare both the PWS and the QASP (KnowNet, 2005). On the other 
hand, both the government and contractor personnel agreed that the QASP and 
surveillance techniques could be viewed as subjective. As the literary research has 
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shown, “there is nothing wrong with the subjective measurement of a service as long as it 
is done in accordance with a reasonable, fair and consistently applied standard” 
(KnowNet, 2005). 

The TACOM IT Contractor stated that surveillance should be done on a bell- 
shape curve basis. This would ensure that the highly visible, critical elements are running 
smoothly at all times. The contractor could then focus their performance on these 
elements to ensure the IT infrastructure is being maintained to avoid downtime and/or 
any degradation to systems/equipment. 

Still, both the government and contractor management interviewed agreed that 
surveillance could be viewed negatively. For example, the previous IT PBSA 
surveillance team focused their results on how much money could be deducted from the 
contract instead of the desired objective of providing the highest quality service to their 
customers. 

As real world experiences and literary review have shown, the contracting officer 
is often tasked with developing the team relationship between the government and the 
contractor. This often requires the Contracting Officer to assume the role of an arbitrator 
to ensure that both the government and contractor are being treated equitably. 

In the case of the most recent IT PBSA, the warranted Contracting Officer stated 
that they were even involved in Alternate Dispute Resolution (ADR) as a means of 
resolving differences between the contractor and the government technical personnel. 
The ADR approach was used to resolve differences between the contractor and the 
government technical personnel. The differences ranged from measurable standards, an 
acceptable quality level, incentives, etc. All parties surveyed indicated that this situation 
was partly attributed to the fact that the government lost the A-76 competition. 
Consequently, the surveillance team was perceived to be too harsh and did not always 
appear to conduct themselves with the utmost integrity. 

On the other hand, the contractor personnel stated that, in their opinion, the 
QASPs are not necessarily a good tool to gauge the contractor’s performance. They felt 
that the same surveillance could be accomplished by scheduling weekly, biweekly, 
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monthly, or quarterly In-process Review (IPR) meetings. These IPRs can be structured 
with an agenda and can provide a forum for both the government and the contractor to 
gauge if the intended outcomes are being accomplished and, if not, why. Additionally, 
these IPRs can be used as a forum in establishing and building the relationship between 
the two parties. 

Consequently, it is imperative that the team be established from the onset, and the 
team should consist of technical personnel, which will eventually have the responsibility 
to perform contractor surveillance. When a team member has to be replaced, the new 
team member should receive PBS A training prior to joining the team. The team member 
should possess technical “knowledge, experience, and insight into what needs to happen 
next, and what is expected during contract perfonnance. Contract award is not the final 
measure of success. Effective and efficient contract perfonnance that delivers a solution 
is the goal. The team should stay together to see that end reached”(Seven steps to PBSA, 
2005). 

C. SUMMARY 

In conclusion, this chapter provided a comparison and analysis of the literary 
review discussed in Chapter II, with the surveys and interviews discussed in Chapter III. 
The research showed that the government personnel lack the training and expertise 
needed in order to clearly define their requirements in order to perform adequate 
contractor surveillance. Additionally, survey results show that PBSA members need to 
have an understanding of their roles and responsibilities in any IT PBSA (Seven Steps to 
PBSA, 2005). The surveys indicated that technical personnel felt that acquisition 
personnel lack knowledge in the IT process, which can hinder their ability to recommend 
the correct acquisition strategy. Additionally, technical personnel’s lack of knowledge 
regarding the acquisition process can cause problems when it comes to contractor 
surveillance. Therefore, technical personnel who will be responsible for monitoring the 
contractor’s perfonnance need to understand their responsibilities long before the 
contract is awarded (Seven Steps to PBSA, 2005). Furthermore, Chapter V will provide 
the research conclusions, recommendations, and areas for further research. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

As stated in Chapter I, the purpose of this Joint Applied Project professional 
report is to analyze IT performance based service acquisitions within the U.S. Army Tank 
Automotive and Armaments Command (TACOM) and to evaluate what performance 
measurements are being used to gauge the contractor’s effectiveness against the 
performance measures stated in the contract. In addition, this research will evaluate 
incentives, as well as penalties used, to incentivize the contractor to provide the level of 
service envisioned by the Government. 

In conclusion, the purpose of this research will analyze how the development of 
performance measures, incentives, and penalties affect the ability of the surveillance team 
to adequately administer the contract. This information will be used to develop and post 
a perfonnance measurement guide on the TACOM Community website to provide all 
contracting personnel with means to develop performance measures for IT PBSAs. 

In order to accomplish this research, the following questions were developed: 

1. To what extent are performance-based measures being used to manage 
TACOM IT PBSAs? 

2. How clear and applicable are PBSA contracting policies and procedures, 
e.g., statements of work (SOWs), performance work statements (PWS), and statements of 
objectives (SOOs)? 

3. What are the incentives and penalties affecting PBS contracting? 

4. What are ways to improve U.S. Army, TACOM-RI, PBSAs, including 
methods promulgating web-based guidance to practitioners? 

This chapter will discuss the research questions and provide conclusions, 
recommendations, and areas for further research based on the analysis performed during 
this research. 
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B. CONCLUSIONS 

In response to the first question evaluated under this research which asked, “To 
what extent are performance-based measures being used to manage TACOM IT PBSAs,” 
the research indicated that nearly all the acquisitions contained performance measures. 
However, the research showed that the performance measures did not necessarily yield 
the desired outcomes. In addition, the research of both governmental guidance and best 
commercial practices showed that these perfonnance measures should be kept to a 
minimum in order to adequately perform contract surveillance. 

However, in reality, the research showed that often the PWS, SOO, QASP, etc., 
included too many performance standards. This contributed to the lack of incentives to 
motivate the contractor to exceed expectations, especially on firm fixed price type 
contracts. The research showed that contractors would often zero in on the areas where 
they can gain the most profit at the expense of other tasks required under the contract. 
Although the perfonnance measures are included in the PBSA, they are not necessary 
yielding the desired outcomes. This can be attributed to the PBSA team’s lack of 
experience and training regarding PBSA. 

Additionally, the literary review on PBSA, PWS, and QASP indicates that 
performance measures should begin with the definition of the requirement; the research 
showed this was not always being done. This cannot be accomplished unless the PBSA 
team has a clear definition of the requirements. To accomplish this task, the PBSA team 
must continuously work with the stakeholders, i.e., customer(s) to ensure that they have a 
clear understanding of the desired outcomes. Consequently, it is essential that the team 
members remain consistent throughout the PBSA (Seven Steps to PBSA, 2005). 

In response to the second question, the objective of this research was to find out if 
the current policies and procedures regarding PBSAs are clearly identified and applicable 
for this type of acquisition technique. This includes documents such as statements of 
work (SOWs), performance work statements (PWS), and statements of objectives 
(SOOs). 
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The literary review conducted in Chapter II indicates that there are many PBS A 
guidebooks and reference sites which can be found on the internet. However, as part of 
the research into PBSA, it was discovered that there is some confusion regarding what 
topic to look for, i.e., is the acquisition under PBC, PBSC, and/or PBSA. In fact, the 
OFPP Interagency Task Force has issued recommendations to clear up this confusion. 
The recommendations include adding definitions to FAR Part 2 for PWS, QASP, SOO, 
etc. In addition, this report rescinded out of date guidance and the OFPP Best Practice 
guide of 1998 in favor of the Seven Steps to PBSA guide (PBSA Contracting for the 
Future, 2003). 

As discussed in Chapter II - Literary Review, this guide provides seven 
comprehensive steps along with references in performing any PBSA. Furthermore, the 
KnowNet website at http://knownet.hhs.gov/aboutKnowNet.htm provides a handy 
desktop reference and is broken out into the various acquisition phases, beginning with 
the formulation of the PBSA team through contract completion and close out. 

Another resource is the Department of Defense Education Activity (DoDEA) 
customer guide, which provides non-acquisition personnel with guidance to Federal 
acquisition systems (DoDEA, 2005). Consequently, there are plenty of reference 
materials available to assist the PBSA team with any PBSA. Even though there is 
guidance on PBSA, each team member should have PBSA training. This will assist the 
team in being able to evaluate and narrow down their guidance material specifically for 
their particular type of acquisition. 

The third question used in this research project investigated the incentives and 
penalties affecting PBS contracting and found that there are many types of incentives 
available, both non-monetary and monetary. The research showed that often times, the 
PBSA team reviews the context at FAR 37.6 and uses firm fixed pricing as a technique to 
incentivize the contractor to perform the required services. Based on the surveys, 
interviews, and literary review, this can be attributed to the PBSA team’s lack of 
knowledge and experience in PBSA. Additionally, the team may lack the appropriate 
training to be able to identify what added value incentives can bring in contractor 
surveillance. 
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To conclude, the fourth question used in this research project was previously 
addressed in Chapter IV. The project reviewed the information provided under Chapter 
IV, Analysis and Findings, which indicated that there are various disconnects and 
discrepancies that need to be addressed in any PBSA. Chapter IV also provided various 
ways that a team can improve U.S. Army and TACOM-RI PBSAs, to include methods 
promulgating web-based guidance to the entire acquisition workforce. These methods 
include the following: 

1. Establishing the PBSA Team 

A literary review of Commercial Best Practices and government sources has 
indicated that PBSA teams should be fonned early in the acquisition process. The real 
world infonnation obtained in Chapter III indicates this is not what is currently 
happening in TACOM acquisitions. In addition, the research indicated that there is a lack 
of consistency in team membership, which is inconsistent with the principles stated in the 
Seven Steps of PBSA. 

The team must have senior management support to ensure any PBSA is 
successful. This support includes a commitment that the PBSA team who develops the 
PWS is that same team that provides surveillance after contract award. 

Additionally, the research shows that many of the PBSA team members have not 
received sufficient training on PBSAs to be an effective team member. Yet, there is no 
indication that training has been a requirement in the past or will be in the future for 
PBSA team membership. To be an effective member of the PBSA team, all team 
members should have documented proof that they have received training in the PBSA 
process and have a basic understanding of the acquisition process. 

Furthermore, the research has shown that the acquisition workforce does not 
necessarily have the experience or the knowledge base required to perform a PBSA. 
Therefore, the acquisition workforce must receive training on PBSAs and, if possible, be 
allowed to participate on a PBSA before they are assigned the responsibility of a PBSA. 
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The TACOM organization must recognize and support the PBSA team in the challenges 
they may face throughout the PBSA process. Only then will the team feel empowered 
enough to possibly think outside the box. 

2. Defining the Requirement 

The research indicates that more attention needs to be given to defining the 
requirement. The team should assist the customer in identifying their requirements in 
terms of outcomes instead of stating how to perform the services required. 

Identification of the requirement should be associated with the organization’s 
mission and must be measurable. However, in order for this to occur, the PBSA team 
and the organization must move away from doing business as usual. This will involve 
significant changes in the way TACOM currently conducts their PBSAs, and they cannot 
be successful without conducting adequate market research. 

3. Conducting Market Research 

All acquisitions, including PBSAs, are required to have documentation that shows 
adequate market research was conducted. Yet, many in the acquisition workforce and 
PBSA team members do not know what constitutes adequate market research or how to 
go about it. 

Acquisition personnel are usually brought into the PBSA acquisition process after 
the requirement has been identified. As a result, market research is usually performed 
after the customer’s minimum needs have been recognized. 

The responsibility of conducting market research was designed to be 
accomplished by the Program Manager as they have the market expertise to conduct 
adequate market research; but the requirement to conduct the market research has been 
passed to the contracting officers/ contract specialist (1102 s). 

Traditionally, the 1102s do not have the market expertise nor the background and 
experience to be market analysts. As a result, 1102s are now expected to assume a 
different role and become market experts. Until 1102s are able to get more experience in 
becoming experts in the markets of the services they purchase, adequate market research 
will not be completed. 
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By conducting adequate market research, the PBSA team will be able to 
determine the appropriate contract type, terms and conditions, length of the contract, 
incentives, and industry practice and standards. 

Without adequate market research, the best solutions available to meet the 
customer requirements may not be considered. This results in the customer not being 
satisfied and the organizations mission not being met. 

The Seven Steps to PBSA states the team should use commercial standards 
whenever possible, which could assist the team in formulating their perfonnance 
standards and metrics. Additionally, if there are industry standards available, then all 
potential offerors should already know what the expected performance is in any resulting 
contract. 

In addition, market research can assist the team in selection of contract type and 
incentives for the PBSA. 

Still the team and the organization should place more emphasis on perfonning 
market research throughout the process (Seven Steps to PBSAs, 2005). 

4. Solicitation Development - SOW, Metrics, Incentives, etc. 

It is essential that the PBSA team write the PWS in relevant and measurable 
terms. Additionally, the PWS must be written in terms of what the desired outcomes are 
and not what or how the services were previously performed. This is a cultural change 
for the technical personnel who once provided the services. Often the COR, COTR, or 
the QAE previously performed the same function and are now responsible for monitoring 
the contractor performance. Consequently, it is imperative that PBSA team applies sound 
business judgment when writing the PWS to ensure that the government provides all 
offerors with clear, concise, and measurable standards for the required services. 
Otherwise, contractors may fail to submit alternative approaches to perform the same 
services, which could result in a lower cost to the customer or a higher-level of service. 

5. Measurable Performance Standards or Metrics 

Although the literary review on PBSA provided information and references 
regarding metrics, it is unclear that the Government personnel surveyed had any 
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knowledge or experience in selecting and evaluating metrics for any PBSA. The research 
indicated that the PBSA team should use existing commercial quality standards, if 
applicable. These standards should be identified during market research if done properly 
(Seven Steps to PBSA, 2005). 

Additionally, a Quality Assurance Plan (QAP) should be requested as part of the 
acquisition process. It ensures that the government receives the required services under 
any resulting contract. This plan provides detailed steps, which the contractor must take 
to make sure that they perform in accordance with the performance standards as stated in 
the contract. 

The QAP sets the "what," "how," and "how often" of quality assurance, and helps 
the offeror determine the level of resources needed to adequately staff the project. It also 
reflects the value of each performance requirement as a percentage of the overall 
contract, so the offeror can focus his resources in the appropriate areas” (Rogin, 2001). 

However, it does not have to be made part of the resulting award. Usually after 
award, the contractor will submit a quality control plan (QCP). The QCP must be 
approved soon after the contract is awarded and is often modified as needed to cover 
unanticipated changes or new technologies. As a result, the PBSA team must conduct 
adequate market research to assist them in identifying the appropriate metrics needed for 
each PBSA. 

6. Incentives and Remedies 

The literary review indicated that incentives could be monetary or non-monetary 
and negative or positive. However, it is unclear when or what incentives are appropriate 
for each PBSA reviewed. The incentives are used to motivate contractor performance. 
Many of the acquisitions that were reviewed during this research project were Finn Fixed 
Price (FFP). Often the FFP contracts are viewed to have a negative incentive, and the 
focus is on deductions for non-performance instead of motivating the contractor to 
exceed contract expectations. Consequently, the PBSA team and acquisition personnel 
have not taken into consideration the value added importance that incentives can bring in 
contractor surveillance. 
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7. Awarding the Contract 

As stated in the Seven Steps to PBSA, the “ key to selecting the right contractor is 
to structure the acquisition so that the government describes the problem that needs to be 
solved and vendors compete by proposing solutions.” This will assist the PBSA team in 
selecting a contractor who offers a quality solution to the PWS and the contractor’s 
proposal will then “becomes a true discriminator in best-value evaluation” (Seven Steps 
to PBSA, 2005). 

However, the research of the real world experiences indicates that best value 
acquisitions often are selected by the lowest price offered. In conclusion, the PBSA team 
needs to evaluate the PBSA to see if best value techniques are appropriate and, if so, that 
they have the knowledge and experience to select the appropriate evaluation factors 
which will yield the best value to the government. Thus, if the contractor represents the 
best value to the government, then the team should be able to monitor the contractor’s 
performance based on the infonnation provided during the solicitation. 

8. Contract Surveillance 

As GAO report 05-274 stated, DoD has not adequately performed contractor 
surveillance (GAO report 05-274, 2005). Yet this is not the case based on the results 
from survey and interviews performed for this research project. The QAEs perform 
contractor surveillance either on a daily, weekly or monthly basis. 

The research also indicated that the QAEs often sought ways to reduce the 
contract amount due to inadequate and/non-performance issues. The QASP contained 
too many tasks that were being measured; however, they might have been relevant to the 
overall performance of the contract, or they may have decreased the significance of each 
task to where the contractor picks and chooses which task to perform. 

Consequently, the QAEs and the PBSA team need to remember the importance of 
obtaining desired outcomes and meeting organizational mission and goals when writing 
the PWS and the QASP. This will ensure that customer’s needs are being met. 
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c. 


RECOMMENDATIONS 


In order to be able to increase the number of PBSAs being done in the 
organization and to ensure the ability of the PBS A team to effectively perform 
surveillance on the resulting contracts, the research indicates that DoD’s culture must 
change. The workforce, including management, must have a clear focus on the reasons 
why PBSAs are important and how they can help the PBSA team ensure that the PBSA 
will meet the performance goals in order to accomplish the organization’s overall 
mission. 

Additionally, the PBSA team must let the contractor provide solutions to the 
stated problem and let the contractor be responsible for contract performance. If the 
government directs the contractor on how to perfonn the services, then, in essence, 
“they've taken responsibility away from the contractor and onto themselves," (Mathers, 
2001 ). 

To help ensure contractor provides the level of performance that the government 
has envisioned, the contractor must become a part of the team after contract award and 
must be treated as such. This is another cultural change for the government acquisition 
workforce. 

Consequently, the team takes on a different role of working with the contractor 
and also monitoring their perfonnance. The key is to build on the relationship with the 
contractor and to find ways to motivate them to exceed expectations; thus, satisfying the 
customers needs while meeting the organizational goals. 

Consequently, it is important the PBSA team be established from the onset of the 
acquisition and remains consistent throughout the acquisition process to contract 
closeout. 

1. Team Consistency 

As identified in our literary review regarding PBSA, both industry and 
government resources have stated that in order to adequately perform contractor 
surveillance there must be consistency in the team membership. Organizations must 
ensure that the team who initiates the acquisition should also be the team that performs 
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contractor surveillance. After all, they are the ones that have the knowledge to judge 
whether the performance standards are being met or if changes are in order. 

2. Conduct Adequate Market Research 

The PBSA team needs to be held accountable for conducting adequate market 
research. They need to understand that market research is not a one-time effort and not 
the responsibility of the contracting officer. Additionally, they need to perform market 
research in conjunction with assisting the customer(s) in identifying their minimum 
needs. Therefore, open communication is required with all stakeholders involved in the 
PBSA. 

The literary review states that the team can obtain assistance from private industry 
regarding market research. However, this does not solve the problem of knowing when 
and how to perfonn market research. Market research is just not the identification of 
solutions for the problem; it also includes terms and conditions, length of the contract, 
incentives, industry practices and standards. 

The Seven Steps to PBSA states, the team should use commercial standards 
whenever possible, which could assist the team in formulating their perfonnance 
standards and metrics. Additionally, if there are industry standards available, then all 
potential offerors should already know what the expected performance is in any resulting 
contract. 

In addition, market research can assist the team in selection of contract type and 
incentives for the PBSA. The PBSAs reviewed at TACOM were firm fixed price (FFP) 
type contracts. These types of contracts place the total risk on the contractor and should 
incentivize them to hold down costs. However, this type of vehicle may not be the 
correct vehicle to use on PBSA. The team needs to make sure that the acquisition 
contains incentives which will allow the contractor to propose innovative approaches and 
ultimately provide savings to the government. Consequently, the incentives included in 
the acquisition must be equivalent to the risk and complexity of the PBSA. 
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3. Knowledge of Monitoring Tools 

The literary review on PBSA revealed that there are no templates for PWS, 
QASP, etc. (Guideline to PBSA, 2001). Still, the PBSA team should be able to obtain 
lessons learned and copies from present or past acquisitions to assist them in drafting 
these documents from various resources, such as, KnowNet, Seven Steps to PBSA, etc. 
This infonnation can help the COR, COTR, and/or the QAE with the use of surveillance 
tools such as QASP and Acceptable Quality Level (AQL). After all they are responsible 
for ensuring that the contractor is meeting the desired outcomes. 

The QASP is an essential piece of any PBSA. Yet there is no clear and concise 
guidance on how to develop one. In order to achieve this, the PBSA team should make 
sure that the QASP reflects the contract objectives. In addition, the team needs to make 
certain that the level of surveillance reflects the critical level, risk, dollar value, etc., of 
the required services (Acquisition Advisory Panel Performance-Based Contracting 
Subpanel Report, 2005). 

The DOE Acquisition guidebook states “The QASP concentrates on the level of 
performance and not the method for achieving it” (DODEA, Customer Guide, 2005). 

4. Requiring Training on PBSAs 

All of the PBSA team members should receive training before they can participate 
in any PBSA. The team leader should have documented proof for each team member of 
the training they have received. 

There are several excellent training sources available, through the Defense 
Acquisition University. Training can often be brought on-site if the requirement exists or 
the person can be sent to a training facility. In addition, there are several excellent on line 
web sites, which can assist the team with the PBSA. The KnowNet is comprised of 
different modules and provides frequently asked questions, along with reference material. 
Also, the Seven Steps to PBSA provides an extensive and comprehensive guidebook 
regarding PBSAs. Additionally, the Guide to PBSA, dated 2000, provides the guiding 
principles required for any PBSA. Consequently, the resources available to the team are 
abundant; it just takes time to research and time is not always available. 
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D. AREAS FOR FURTHER RESEARCH 

1. Does the A-76 acquisition process hinder competition and cooperation 
on PBSAs? 

Although the A-76 acquisition process has been streamlined, it is still very 
confusing. 

2. Is the Government effectively using Quality Assurance Surveillance 
Plan (QASP) on PBSA and are the plans being effectively monitored? 

In testimony before the Panel, Ms. Barbara S. Kinosky from Centre Consulting 
stated, “Government needs to learn not to create overly burdensome surveillance plans 
that will ultimately create a bureaucracy of contractors, monitoring contractors for 
compliance, only evaluate what is necessary to accurately measure success” (Ensuring 
the Effective and Appropriate Use of PBC. 2005). 

3. Is the traditional 1102s receiving adequate training to assist them in 
performing PBSA? Should their expertise become more like that of a Market 
Analyst instead of just a manager of the acquisition process? What training can be 
provided to teach our 1102s to become market analysts? 

Most service acquisitions do not usually have a specific Program Manager; 
therefore it is usually the contracting officer responsibility to lead the PBSA. This role is 
new to the traditional 1102, and they may not necessarily possess the knowledge and/or 
training to perform these duties. Traditionally, the contracting officer or 1102 has not had 
to be a market analyst, but with requirements of a PBSA, they must become market 
analysts and become technically familiar with the type of services required. In fact, this 
dilemma has lawmakers so concerned that the recently approved National Defense 
Authorization Act for Fiscal 2006 provides “for DoD officials to increase the number of 
acquisition employees by 15 percent by fiscal 2009.” Furthermore, the bill also calls for 
“a focus on hiring workers with skills in specific areas, such as writing perfonnance 
based statements of work..(Miller, 2005). Consequently, the current workforce should 
also be trained to perform these same duties and to act more like a market analyst. 

4. How can the Government ensure that the team remains consistent 
throughout the acquisition process? 

The FAR identifies the roles and responsibilities of the PBSA team, as well as 
stating that the team should be empowered to make decisions. However, it fails to state 
that the team members should be committed to the acquisition even after contract award. 
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As the research indicated, this is crucial for the PBSA team to be able to effectively 
monitor contractor performance. Yet, how can the government ensure that the team 
remains consistent? 

Another question that arises is how can we ensure that the acquisition team 
adequately perfonns market research? 

Although the FAR requires that each acquisition include adequate market 
research, it is still being left to the acquisition team. Team members are often not 
familiar with market research techniques; therefore, they are not sure how they can assist 
the acquisition personnel in perfonning market research. 

E. SUMMARY/CONCLUSION 

This chapter concludes the research perfonned regarding surveillance of PBSAs. 
As previously stated in this chapter, the research provided a summary of the conclusions 
uncovered during the research project. Additionally, it included recommendations which 
can assist the acquisition workforce and the PBSA team with all different stages of a 
PBSA. Finally, this chapter also included areas for further research regarding PBSAs. 
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